Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.93275826s FAIL (2025-10-07 14:48:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.932789749s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.528336633s FAIL (2025-10-07 14:48:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.528371388s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 52.024981779s FAIL (2025-10-07 14:47:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 52.025014102s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.207262924s FAIL (2025-10-07 12:50:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.20729211s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.056253841s FAIL (2025-10-07 12:49:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.056302283s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.88227303s FAIL (2025-10-07 12:48:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.882306504s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.803856088s FAIL (2025-10-07 12:47:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.803886827s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.228016848s FAIL (2025-10-07 12:46:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.228048536s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.886743358s FAIL (2025-10-07 12:45:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.886779165s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 9.402851502s FAIL (2025-10-07 12:45:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 9.402885196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.838722272s FAIL (2025-10-07 12:45:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.838764432s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.998725456s FAIL (2025-10-07 12:44:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.998755905s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.548110404s FAIL (2025-10-07 12:44:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.548259389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.97986376s FAIL (2025-10-07 12:43:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.97989538s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.123797778s FAIL (2025-10-07 12:43:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.123833135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.983236986s FAIL (2025-10-07 12:43:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.983265931s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.749300687s FAIL (2025-10-07 12:42:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.749336516s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.024884412s FAIL (2025-10-07 12:42:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.024919761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.324518775s FAIL (2025-10-07 12:41:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.324556687s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.061699568s FAIL (2025-10-07 12:41:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.061743852s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.991442948s FAIL (2025-10-07 12:41:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.991478395s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.37451614s FAIL (2025-10-07 12:40:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.374543681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.895276752s FAIL (2025-10-07 12:40:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.895313703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.071297577s FAIL (2025-10-07 12:40:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.071329818s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.498235755s FAIL (2025-10-07 12:39:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 7.498268667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.992612799s FAIL (2025-10-07 12:39:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.992650119s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.487084179s FAIL (2025-10-07 12:39:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.487113895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.14286882s FAIL (2025-10-07 12:38:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.142892236s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.444581966s FAIL (2025-10-07 12:38:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.444617734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 11.371879374s FAIL (2025-10-07 12:38:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 11.371901216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.106911331s FAIL (2025-10-07 12:37:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.106946268s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.141995489s FAIL (2025-10-07 12:37:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.142064881s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.101945717s FAIL (2025-10-07 12:37:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.101984039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.006234494s FAIL (2025-10-07 12:36:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 7.006271075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.846247011s FAIL (2025-10-07 12:36:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.846279433s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 9.147451476s FAIL (2025-10-07 12:35:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 9.147482114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.933094659s FAIL (2025-10-07 12:35:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.933131059s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.059552653s FAIL (2025-10-07 12:34:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.059589442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.271739979s FAIL (2025-10-07 12:34:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.271772009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.648743123s FAIL (2025-10-07 12:34:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.648785023s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.884788547s FAIL (2025-10-07 12:33:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.884825438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.318395248s FAIL (2025-10-07 12:33:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.318429915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.891112604s FAIL (2025-10-07 12:33:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.891140897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.287006953s FAIL (2025-10-07 12:33:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.287052811s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.31767581s FAIL (2025-10-07 12:32:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.317710226s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.006541169s FAIL (2025-10-07 12:32:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.006586125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.189401501s FAIL (2025-10-07 12:31:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.189439521s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.12792145s FAIL (2025-10-07 12:31:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.127954533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.882334739s FAIL (2025-10-07 12:31:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.88236187s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 7.252681079s FAIL (2025-10-07 12:30:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 7.252721746s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)