Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.....X..X..X.....] duration: 24.451619127s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:04:35 UTC) orbiter warning: context deadline exceeded Duration: 24.451646648s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXX] duration: 5.001226664s trace: https://jaeger.dev.storj.io/trace/4345c35aa27fea71 FAIL (2025-12-15 13:03:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001254026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X.....X..X..X....] duration: 14.254912399s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:03:44 UTC) orbiter warning: context deadline exceeded Duration: 14.254943107s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXX] duration: 5.002432048s trace: https://jaeger.dev.storj.io/trace/482a103337209da4 FAIL (2025-12-15 13:03:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002456866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.....X..X..X...] duration: 18.081290736s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:02:55 UTC) orbiter warning: context deadline exceeded Duration: 18.081311014s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXX] duration: 5.000571245s trace: https://jaeger.dev.storj.io/trace/2d96af529ac37530 FAIL (2025-12-15 13:02:36 UTC) orbiter warning: context deadline exceeded Duration: 5.000585754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X.....X..X..X..] duration: 11.591363467s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:01:57 UTC) orbiter warning: context deadline exceeded Duration: 11.591390659s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X.....X..X..X.] duration: 13.285817251s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:01:39 UTC) orbiter warning: context deadline exceeded Duration: 13.285844613s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXX] duration: 5.001354847s trace: https://jaeger.dev.storj.io/trace/625e311ff3004f3b FAIL (2025-12-15 13:01:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001375537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X.....X..X..X] duration: 30.001784677s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:01:21 UTC) orbiter warning: context deadline exceeded Duration: 30.001810826s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.....X..X..] duration: 15.08059939s trace: https://jaeger.dev.storj.io/trace/acc5e676b1715ed FAIL (2025-12-15 13:00:43 UTC) orbiter warning: context deadline exceeded Duration: 15.080620821s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXX] duration: 5.001297279s trace: https://jaeger.dev.storj.io/trace/449e51e7a780f7e8 FAIL (2025-12-15 13:00:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001324129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.....X..X.] duration: 27.369800542s trace: https://jaeger.dev.storj.io/trace/acc5e676b1715ed FAIL (2025-12-15 13:00:28 UTC) orbiter warning: context deadline exceeded Duration: 27.369818485s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [.X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXX] duration: 5.001671631s trace: https://jaeger.dev.storj.io/trace/50d292750c895873 FAIL (2025-12-15 13:00:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001696678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 19.658595006s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:59:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 19.658622569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 17.635939041s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:59:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 17.635990119s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [..X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXX] duration: 5.000748273s trace: https://jaeger.dev.storj.io/trace/57193b9b2791cda2 FAIL (2025-12-15 12:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000772209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X.....X..X] duration: 30.001850003s trace: https://jaeger.dev.storj.io/trace/acc5e676b1715ed FAIL (2025-12-15 12:59:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001863739s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [...X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXX] duration: 5.0015494s trace: https://jaeger.dev.storj.io/trace/7cefdd22db261583 FAIL (2025-12-15 12:58:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001576713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 10.930569662s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:58:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 10.930594358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 13.800806912s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:58:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 13.800820558s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration: 15.411507154s trace: https://jaeger.dev.storj.io/trace/98843e1708f5978 FAIL (2025-12-15 12:58:16 UTC) orbiter warning: context deadline exceeded Duration: 15.411527022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 17.641259701s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 17.641287805s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXX] duration: 5.0011197s trace: https://jaeger.dev.storj.io/trace/3b5302dbff7a2f59 FAIL (2025-12-15 12:57:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001142303s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration: 26.943139897s trace: https://jaeger.dev.storj.io/trace/98843e1708f5978 FAIL (2025-12-15 12:57:32 UTC) orbiter warning: context deadline exceeded Duration: 26.943155356s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 9.139629831s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 9.139663615s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration: 30.001808592s trace: https://jaeger.dev.storj.io/trace/98843e1708f5978 FAIL (2025-12-15 12:57:04 UTC) orbiter warning: context deadline exceeded Duration: 30.001893664s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 10.650423004s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:56:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 10.650455596s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [X....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX] duration: 5.001084884s trace: https://jaeger.dev.storj.io/trace/75f96812d06cd3ae FAIL (2025-12-15 12:56:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001113829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 16.633623489s trace: https://jaeger.dev.storj.io/trace/26b5881583cf47bd FAIL (2025-12-15 12:56:14 UTC) orbiter warning: context deadline exceeded Duration: 16.633652402s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 10.164888793s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:56:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 10.164918159s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXX] duration: 5.000719367s trace: https://jaeger.dev.storj.io/trace/53771d54ee1e962b FAIL (2025-12-15 12:55:41 UTC) orbiter warning: context deadline exceeded Duration: 5.000746299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.078210534s trace: https://jaeger.dev.storj.io/trace/26b5881583cf47bd FAIL (2025-12-15 12:55:26 UTC) orbiter warning: context deadline exceeded Duration: 25.078284774s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......X...................................................] duration: 9.776521001s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:55:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 9.776557641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXX] duration: 5.001148594s trace: https://jaeger.dev.storj.io/trace/5607a113b9e06492 FAIL (2025-12-15 12:55:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001175807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXX] duration: 5.000984984s trace: https://jaeger.dev.storj.io/trace/125e5fe482eafb1f FAIL (2025-12-15 12:54:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001004451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 10.058923717s trace: https://jaeger.dev.storj.io/trace/26b5881583cf47bd FAIL (2025-12-15 12:54:17 UTC) orbiter warning: context deadline exceeded Duration: 10.059052663s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......X..................................................] duration: 9.856488597s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:54:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 9.856517002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.429766232s trace: https://jaeger.dev.storj.io/trace/26b5881583cf47bd FAIL (2025-12-15 12:53:28 UTC) orbiter warning: context deadline exceeded Duration: 9.429793012s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.XXXX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XX] duration: 5.00130351s trace: https://jaeger.dev.storj.io/trace/13617c3e28d5463f FAIL (2025-12-15 12:53:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001355168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 11.922573441s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:53:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 11.922609872s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 16.511532314s trace: https://jaeger.dev.storj.io/trace/3bc9ac09147ab561 FAIL (2025-12-15 12:52:42 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816211C22D8CAF, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 16.511622166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.340544938s trace: https://jaeger.dev.storj.io/trace/26b5881583cf47bd FAIL (2025-12-15 12:52:33 UTC) orbiter warning: context deadline exceeded Duration: 22.340575495s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [X.XXXX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.X] duration: 5.001112657s trace: https://jaeger.dev.storj.io/trace/756cfb43a7a257aa FAIL (2025-12-15 12:52:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001128296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 8.985376707s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:52:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 8.985417495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XX.XXXX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.] duration: 4.234989877s trace: https://jaeger.dev.storj.io/trace/63e679ebfc31abfd FAIL (2025-12-15 12:52:02 UTC) orbiter warning: context deadline exceeded Duration: 4.235019111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00402263s trace: https://jaeger.dev.storj.io/trace/26b5881583cf47bd FAIL (2025-12-15 12:51:53 UTC) orbiter warning: context deadline exceeded Duration: 30.004045964s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 19.421775943s trace: https://jaeger.dev.storj.io/trace/3bc9ac09147ab561 FAIL (2025-12-15 12:51:50 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816211C22D8CAF, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 19.421818574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXX.XXXX....X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX] duration: 5.001683404s trace: https://jaeger.dev.storj.io/trace/63e679ebfc31abfd FAIL (2025-12-15 12:51:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001703551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 9.672767816s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:51:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 9.672812312s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)