Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XX.] duration: 4.744593855s trace: https://jaeger.dev.storj.io/trace/670afeb7e644e9 FAIL (2026-01-12 17:18:59 UTC) orbiter warning: context deadline exceeded Duration: 4.744618371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X..X.....X....................X....................] duration: 16.398696396s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:18:54 UTC) orbiter warning: context deadline exceeded Duration: 16.398720862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX................X......X....................X....] duration: 23.957014324s trace: https://jaeger.dev.storj.io/trace/299caadca7730f0e FAIL (2026-01-12 17:18:52 UTC) orbiter warning: context deadline exceeded Duration: 23.957042978s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XX] duration: 5.000853676s trace: https://jaeger.dev.storj.io/trace/670afeb7e644e9 FAIL (2026-01-12 17:18:26 UTC) orbiter warning: context deadline exceeded Duration: 5.000876309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.X] duration: 5.000985607s trace: https://jaeger.dev.storj.io/trace/4d22b5f00f7457ef FAIL (2026-01-12 17:18:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001006247s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX................X......X....................X...] duration: 14.552027602s trace: https://jaeger.dev.storj.io/trace/299caadca7730f0e FAIL (2026-01-12 17:18:04 UTC) orbiter warning: context deadline exceeded Duration: 14.552050625s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.] duration: 3.757469286s trace: https://jaeger.dev.storj.io/trace/427dc7758dfddffa FAIL (2026-01-12 17:17:48 UTC) orbiter warning: context deadline exceeded Duration: 3.757500274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X..X.....X....................X...................] duration: 16.107513267s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:17:44 UTC) orbiter warning: context deadline exceeded Duration: 16.107539377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX................X......X....................X..] duration: 8.118155417s trace: https://jaeger.dev.storj.io/trace/299caadca7730f0e FAIL (2026-01-12 17:17:36 UTC) orbiter warning: context deadline exceeded Duration: 8.118175735s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [X.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000793191s trace: https://jaeger.dev.storj.io/trace/427dc7758dfddffa FAIL (2026-01-12 17:17:20 UTC) orbiter warning: context deadline exceeded Duration: 5.000824059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X..X.....X....................X..................] duration: 14.641771686s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:16:45 UTC) orbiter warning: context deadline exceeded Duration: 14.641851748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX................X......X....................X.] duration: 16.603317305s trace: https://jaeger.dev.storj.io/trace/299caadca7730f0e FAIL (2026-01-12 17:16:33 UTC) orbiter warning: context deadline exceeded Duration: 16.60334644s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 5.001137255s trace: https://jaeger.dev.storj.io/trace/2399577693685235 FAIL (2026-01-12 17:16:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001163143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX................X......X....................X] duration: 30.001105885s trace: https://jaeger.dev.storj.io/trace/299caadca7730f0e FAIL (2026-01-12 17:16:13 UTC) orbiter warning: context deadline exceeded Duration: 30.001129168s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 5.001294988s trace: https://jaeger.dev.storj.io/trace/3b3cf4a483863565 FAIL (2026-01-12 17:15:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001316581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX................X......X....................] duration: 16.274516155s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:15:42 UTC) orbiter warning: context deadline exceeded Duration: 16.274529671s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X..X.....X....................X.................] duration: 13.730603762s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:15:42 UTC) orbiter warning: context deadline exceeded Duration: 13.730694926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.001576192s trace: https://jaeger.dev.storj.io/trace/14005743d798640c FAIL (2026-01-12 17:15:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001805278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X..X.....X....................X................] duration: 14.092264261s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:14:57 UTC) orbiter warning: context deadline exceeded Duration: 14.092313975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.865245696s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:14:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.865270632s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX................X......X...................] duration: 17.40326532s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:14:47 UTC) orbiter warning: context deadline exceeded Duration: 17.403291129s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.929088446s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:14:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.929126368s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX................X......X..................] duration: 19.422952845s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:14:17 UTC) orbiter warning: context deadline exceeded Duration: 19.422978653s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.00551922s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:14:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 6.005568696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X..X.....X....................X...............] duration: 31.804371499s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:14:00 UTC) orbiter warning: context deadline exceeded Duration: 31.804396679s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 5.00211897s trace: https://jaeger.dev.storj.io/trace/120ca7e9d882b97e FAIL (2026-01-12 17:13:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002142576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.851759328s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:13:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.851788804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X..X.....X....................X..............] duration: 18.196507819s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:13:25 UTC) orbiter warning: context deadline exceeded Duration: 18.196532668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX................X......X.................] duration: 21.091168764s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:13:11 UTC) orbiter warning: context deadline exceeded Duration: 21.091195876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.827655736s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:13:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.82768427s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 5.001142996s trace: https://jaeger.dev.storj.io/trace/400ae1826e5356dc FAIL (2026-01-12 17:13:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001168044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X..X.....X....................X.............] duration: 15.463596506s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:12:38 UTC) orbiter warning: context deadline exceeded Duration: 15.463619039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.951217689s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:12:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.951246494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 5.000910003s trace: https://jaeger.dev.storj.io/trace/7ea94418cc75d6af FAIL (2026-01-12 17:12:35 UTC) orbiter warning: context deadline exceeded Duration: 5.00093486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX................X......X................] duration: 19.932261812s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:12:30 UTC) orbiter warning: context deadline exceeded Duration: 19.932289013s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.76150742s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.761532668s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX................X......X...............] duration: 17.484915111s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:11:50 UTC) orbiter warning: context deadline exceeded Duration: 17.484938104s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X..X.....X....................X............] duration: 16.193652761s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:11:25 UTC) orbiter warning: context deadline exceeded Duration: 16.193702064s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 5.001105826s trace: https://jaeger.dev.storj.io/trace/1def4b9f59e92251 FAIL (2026-01-12 17:11:22 UTC) orbiter warning: context deadline exceeded Duration: 5.001146964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.822904475s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:11:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.8229278s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.001102718s trace: https://jaeger.dev.storj.io/trace/73770f0338e91f56 FAIL (2026-01-12 17:10:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001126433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX................X......X..............] duration: 16.397290328s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:10:35 UTC) orbiter warning: context deadline exceeded Duration: 16.397315847s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X..X.....X....................X...........] duration: 15.258402547s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:10:21 UTC) orbiter warning: context deadline exceeded Duration: 15.258428456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX................X......X.............] duration: 17.416056208s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:10:05 UTC) orbiter warning: context deadline exceeded Duration: 17.416082829s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X..X.....X....................X..........] duration: 21.071804179s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:10:03 UTC) orbiter warning: context deadline exceeded Duration: 21.071833166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 3.945069725s trace: https://jaeger.dev.storj.io/trace/213d074e27145df5 FAIL (2026-01-12 17:09:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.945094424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.259126792s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:09:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 6.259150756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.0264688s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 17:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 6.026488018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X...................X..X.....X....................X.........] duration: 16.651688152s trace: https://jaeger.dev.storj.io/trace/4cc6392ff45d2e58 FAIL (2026-01-12 17:09:29 UTC) orbiter warning: context deadline exceeded Duration: 16.651712647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX................X......X............] duration: 14.848188469s trace: https://jaeger.dev.storj.io/trace/5a072cc19593affb FAIL (2026-01-12 17:09:26 UTC) orbiter warning: context deadline exceeded Duration: 14.84821525s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)