Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.821979313s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:31:54 UTC) orbiter warning: context deadline exceeded Duration: 5.822005783s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.698397647s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:30:42 UTC) orbiter warning: context deadline exceeded Duration: 5.698457833s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.1784398s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:30:06 UTC) orbiter warning: context deadline exceeded Duration: 6.178468935s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.228046758s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:29:08 UTC) orbiter warning: context deadline exceeded Duration: 6.228078548s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.92440386s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:28:39 UTC) orbiter warning: context deadline exceeded Duration: 5.924428457s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.92862687s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:28:09 UTC) orbiter warning: context deadline exceeded Duration: 5.928656696s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.148114169s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:27:11 UTC) orbiter warning: context deadline exceeded Duration: 6.148143555s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.878567468s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:26:52 UTC) orbiter warning: context deadline exceeded Duration: 5.878594298s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.945773722s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:26:32 UTC) orbiter warning: context deadline exceeded Duration: 5.945797227s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.114776842s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:26:11 UTC) orbiter warning: context deadline exceeded Duration: 6.115119314s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.973815465s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:25:21 UTC) orbiter warning: context deadline exceeded Duration: 5.973846415s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.994723038s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:24:10 UTC) orbiter warning: context deadline exceeded Duration: 5.994764396s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.930195977s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:23:20 UTC) orbiter warning: context deadline exceeded Duration: 5.930224511s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.783477447s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:22:56 UTC) orbiter warning: context deadline exceeded Duration: 5.783501352s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.068720212s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:21:46 UTC) orbiter warning: context deadline exceeded Duration: 6.068746772s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.773886566s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:20:39 UTC) orbiter warning: context deadline exceeded Duration: 5.773912325s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.665505297s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:19:34 UTC) orbiter warning: context deadline exceeded Duration: 5.665541797s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.264812633s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:19:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 6.265001903s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.679693346s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:19:12 UTC) orbiter warning: context deadline exceeded Duration: 5.679738783s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.788720932s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:18:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.78874616s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.817558212s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:18:16 UTC) orbiter warning: context deadline exceeded Duration: 5.817585965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.966331834s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:17:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.966362413s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.0609927s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:17:11 UTC) orbiter warning: context deadline exceeded Duration: 6.061058546s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.142582611s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:16:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 6.142612097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.918917381s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:16:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.918947208s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.946307146s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:16:09 UTC) orbiter warning: context deadline exceeded Duration: 5.9463523s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 14.622916463s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 14.622956508s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.781958314s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:15:13 UTC) orbiter warning: context deadline exceeded Duration: 5.781986057s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.932350849s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:14:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.932379183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.748698032s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:14:42 UTC) orbiter warning: context deadline exceeded Duration: 5.748727238s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.785430187s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:14:20 UTC) orbiter warning: context deadline exceeded Duration: 5.785458069s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......X....................................................] duration: 5.868069325s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:14:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.868095094s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.82127524s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:13:18 UTC) orbiter warning: context deadline exceeded Duration: 5.82130609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......X...................................................] duration: 5.827636132s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:13:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.827662833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......X..................................................] duration: 6.193322018s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:12:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 6.193351374s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.775201716s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:12:03 UTC) orbiter warning: context deadline exceeded Duration: 5.775227506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......X.................................................] duration: 5.998831011s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:11:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.998858946s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.738890255s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:11:16 UTC) orbiter warning: context deadline exceeded Duration: 5.738917997s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 5.965527563s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:10:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.965556168s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.785693451s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:10:07 UTC) orbiter warning: context deadline exceeded Duration: 5.785726133s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.838388157s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:09:51 UTC) orbiter warning: context deadline exceeded Duration: 5.838411412s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X......X...............................................] duration: 5.930836938s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:09:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.93086446s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.997710465s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:09:17 UTC) orbiter warning: context deadline exceeded Duration: 5.997747636s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX....X......X..............................................] duration: 5.892219652s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:09:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.892434091s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001415007s trace: https://jaeger.dev.storj.io/trace/47a98ef3feebc2d FAIL (2025-12-17 18:08:55 UTC) orbiter warning: context deadline exceeded Duration: 30.001445074s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX....X......X.............................................] duration: 6.084742463s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:08:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 6.084768182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX....X......X............................................] duration: 5.810373192s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:07:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.810402327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX....X......X...........................................] duration: 5.760363985s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:06:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.760417837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX....X......X..........................................] duration: 6.257678536s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 6.257701959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX....X......X.........................................] duration: 5.799237097s trace: https://jaeger.dev.storj.io/trace/7f1a7a216ca8811a FAIL (2025-12-17 18:06:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:35:45.345011074 (290h6m45.216094109s) [upload=254348]: Starting upload Dec 17 17:35:45.345096045 (290h6m45.21617909s) [upload=254348]: Done starting upload Dec 17 17:35:45.345207368 (290h6m45.216290403s) [upload=254348]: Uploading segments... Dec 17 17:35:45.346123813 (290h6m45.217206848s) [upload=254348]: Got next segment. Inline: false Dec 17 17:35:45.346271774 (290h6m45.217354809s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:35:45.346277275 (290h6m45.21736031s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:35:45.620395346 (290h6m45.491478391s) [seg_pos={0 0}, upload=254348]: Begin upload segment... Dec 17 17:35:45.620462314 (290h6m45.491545349s) [seg_pos={0 0}, upload=254348]: Done begin upload segment. Dec 17 17:35:45.620500937 (290h6m45.491583972s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620508782 (290h6m45.491591817s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.620933511 (290h6m45.492016547s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.621768772 (290h6m45.492851807s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622192799 (290h6m45.493275834s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622442034 (290h6m45.493525069s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.622807169 (290h6m45.493890204s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.623290891 (290h6m45.494373936s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.624627425 (290h6m45.49571047s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.625507291 (290h6m45.496590326s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.627060128 (290h6m45.498143173s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.628501703 (290h6m45.499584748s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.630080720 (290h6m45.501163765s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.631808500 (290h6m45.502891535s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.633046978 (290h6m45.504130013s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:45.635143409 (290h6m45.506226454s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Uploading piece... Dec 17 17:35:46.723180535 (290h6m46.5942636s) [noise=true , address=77.90.31.18:4805 , piece_id=L23NZI2EPN72K356ZSQ6QM5EMXPJKOJHXIAV4N35AEQ4MH6JTP7Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 17 17:35:47.331465593 (290h6m47.202548638s) [upload=254348]: Next returned nil segment Dec 17 17:35:47.331478788 (290h6m47.202561823s) [upload=254348]: Waiting for error group managing segments... Dec 17 17:35:47.608095567 (290h6m47.479178602s) [noise=true , address=145.79.227.32:4836 , piece_id=WPQUMKMSJE4HRUPV63H6ZTXAV4VQFU5A42S53CB5AIKSCGUK56TQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.611604820 (290h6m47.482687865s) [noise=true , address=145.79.227.13:4804 , piece_id=HUXY6OK2W6YK7HQ3EVUJO3NPKSCXGY5XSRCOFELJYPOYF2YF27LA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.623600612 (290h6m47.494683657s) [noise=true , address=145.79.227.24:4583 , piece_id=BWEGLAN6A7D3EARHHRF7G7D6THLYXKRFDPP7UG3TQLU7MEJL67BQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.627163436 (290h6m47.498246481s) [noise=true , address=145.79.227.48:4302 , piece_id=JYOAULCSKKO2CZA2NEDV4HR5JPH3DQGFY7X7TTUYP2F5PGTIEROQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.639184636 (290h6m47.510267681s) [noise=true , address=145.79.227.19:4077 , piece_id=4KZOACXEPNLT6J5JO4MGIRPVVSSUWXEZJTKMYE4SRSUYGE3U25IA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.640394230 (290h6m47.511477275s) [noise=true , address=145.79.227.18:4006 , piece_id=OZCY5XGSGAIMXVVYLCRCM6Y2BTHLSLUKWNYYHKI44YQM6HYFWR2Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.644907354 (290h6m47.515990389s) [noise=true , address=145.79.227.35:4483 , piece_id=4XWWUHZOO6MQRFFZ7SNAYUDWLMIULARR5HDS57DM3N466JRKEWKQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.678645246 (290h6m47.549728281s) [noise=true , address=145.79.227.21:4474 , piece_id=KT5RCVV5OKUIX3CVFFYPZY3LFWCJYTIYJZQRLACTVJHYQBTABPUQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.820124009 (290h6m47.691207054s) [noise=true , address=77.90.31.13:4629 , piece_id=VBNUFJPHT2YEN5RODHSWU3FEOHHO7ZZQCUZ6OLDAIYDB4VZ7UBRQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.860071276 (290h6m47.731154321s) [noise=true , address=77.90.31.10:4517 , piece_id=JCSQB6M2YAUX7KP5HKMF7DOEM756VQLIGIITAMPL2BBJ4HIAUPWA, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.881173629 (290h6m47.752256674s) [noise=true , address=45.91.116.12:4228 , piece_id=DM3YWK2IBORWAMQMHSRYROQ5N2QPQT47KMQ63C34UPBHDPT5GN6Q, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.895806122 (290h6m47.766889167s) [noise=true , address=45.91.116.13:4154 , piece_id=CCFRFYDE4TUR44XH67WSQ6KLKGRL767UGGJ6NV5X56TXAFGMPACQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907327130 (290h6m47.778410175s) [noise=true , address=45.91.116.6:4974 , piece_id=GZL3N452HQ5T33KHRLTR2ROTVSHCGPFEC5S35GD4Q5UGFB67XS2A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: <nil> Dec 17 17:35:47.907351887 (290h6m47.778434922s) [seg_pos={0 0}, upload=254348]: Segment reached optimal threshold of 13 pieces. Dec 17 17:35:47.907632151 (290h6m47.778715196s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907642570 (290h6m47.778725616s) [noise=true , address=45.91.116.7:4009 , piece_id=KAHT3ZXABEQW7QGE6EXAKBTTPJZKIGPA77FCAGX4T6VVHZIBS37A, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907949134 (290h6m47.779032169s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 17:35:47.907957731 (290h6m47.779040776s) [noise=true , address=45.91.116.11:4120 , piece_id=EDSAMJ5A7PIR2VUHLLLNUDGIGYZLYPN7GIXDSH4OYKLA5DDPY7LQ, seg_pos={0 0}, upload=254348]: Long tail cancellation due to optimal threshold Dec 17 17:35:47.907978551 (290h6m47.779061586s) [seg_pos={0 0}, upload=254348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:35:47.908056158 (290h6m47.779139203s) [upload=254348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:35:47.908062871 (290h6m47.779145906s) [upload=254348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:35:48.202450309 (290h6m48.073533355s) [upload=254348]: Done uploading segments... Dec 17 17:35:48.202477340 (290h6m48.073560385s) [upload=254348]: Upload finished. err: <nil> Duration: 5.799256112s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)