Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.182215559s FAIL (2025-10-07 12:19:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.182251868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.365111957s FAIL (2025-10-07 12:19:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.365152965s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.046679031s FAIL (2025-10-07 12:18:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.04671555s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.071145588s FAIL (2025-10-07 12:18:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.071180343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.804087106s FAIL (2025-10-07 12:18:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.80412533s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.308809054s FAIL (2025-10-07 12:18:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.30884338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.034952765s FAIL (2025-10-07 12:17:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.034989486s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.294326702s FAIL (2025-10-07 12:17:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.294366176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.27192846s FAIL (2025-10-07 12:17:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.271965523s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.983652921s FAIL (2025-10-07 12:17:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.983686686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.782046484s FAIL (2025-10-07 12:16:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.782082933s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.329269618s FAIL (2025-10-07 12:16:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.329305147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.742865799s FAIL (2025-10-07 12:15:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.742901486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.732432557s FAIL (2025-10-07 12:15:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.732460179s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.381502136s FAIL (2025-10-07 12:15:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 7.381537363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.808261636s FAIL (2025-10-07 12:14:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.808297334s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.067417914s FAIL (2025-10-07 12:13:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.067446799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.030291733s FAIL (2025-10-07 12:13:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.030319246s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.488523828s FAIL (2025-10-07 12:13:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.488557683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.038634837s FAIL (2025-10-07 12:12:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.038705822s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.167207056s FAIL (2025-10-07 12:12:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.167245218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.807536671s FAIL (2025-10-07 12:11:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.807570185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.468551846s FAIL (2025-10-07 12:11:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.468686552s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.928431385s FAIL (2025-10-07 12:11:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.928466471s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.695004526s FAIL (2025-10-07 12:11:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 7.695037468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.960237891s FAIL (2025-10-07 12:10:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.96027358s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.171418548s FAIL (2025-10-07 12:10:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.171452614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.908086151s FAIL (2025-10-07 12:10:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.908116969s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.210411976s FAIL (2025-10-07 12:09:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.210449056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.97177416s FAIL (2025-10-07 12:09:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.971809187s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 12.026352116s FAIL (2025-10-07 12:09:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 12.026386952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.869780059s FAIL (2025-10-07 12:08:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.869818702s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.214474565s FAIL (2025-10-07 12:08:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.214511797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.944468512s FAIL (2025-10-07 12:08:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.944505372s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.865517356s FAIL (2025-10-07 12:07:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.865547684s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.244468521s FAIL (2025-10-07 12:07:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.244627443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.82011801s FAIL (2025-10-07 12:06:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.820155983s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.817782422s FAIL (2025-10-07 12:06:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.817815506s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.45099373s FAIL (2025-10-07 12:06:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.451033086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.242792922s FAIL (2025-10-07 12:06:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.242853717s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 14.33094214s FAIL (2025-10-07 12:05:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 14.330974582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.476134781s FAIL (2025-10-07 12:04:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.476164909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.959479093s FAIL (2025-10-07 12:03:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.95951449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 13.319520332s FAIL (2025-10-07 12:02:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 13.319554558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.109816796s FAIL (2025-10-07 12:01:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.109850831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.067515201s FAIL (2025-10-07 12:01:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 6.067546892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 20.393548583s FAIL (2025-10-07 12:00:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 20.393575483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002352279s FAIL (2025-10-07 11:59:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 11:59:29.267478555 (334h28m14.729571028s) [upload=294346]: Starting upload Oct 7 11:59:29.267610086 (334h28m14.729702569s) [upload=294346]: Done starting upload Oct 7 11:59:29.267696570 (334h28m14.729789053s) [upload=294346]: Uploading segments... Oct 7 11:59:29.267848139 (334h28m14.729940612s) [upload=294346]: Got next segment. Inline: false Oct 7 11:59:29.267897393 (334h28m14.729989876s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 11:59:29.267904336 (334h28m14.729996819s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 11:59:29.526299052 (334h28m14.988391565s) [seg_pos={0 0}, upload=294346]: Begin upload segment... Oct 7 11:59:29.526361261 (334h28m14.988453744s) [seg_pos={0 0}, upload=294346]: Done begin upload segment. Oct 7 11:59:29.526390957 (334h28m14.988483441s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.529972777 (334h28m14.99206526s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.534716761 (334h28m14.996809244s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.537287506 (334h28m14.999379979s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543370690 (334h28m15.005463173s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.543416878 (334h28m15.005509371s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.550715336 (334h28m15.012807819s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.551986627 (334h28m15.0140791s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.553594939 (334h28m15.015687412s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.556255655 (334h28m15.018348138s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.557511988 (334h28m15.019604471s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.559201445 (334h28m15.021293938s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.561699101 (334h28m15.023791584s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.575205469 (334h28m15.037297943s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.577592746 (334h28m15.039685229s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:29.578218557 (334h28m15.040311101s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Uploading piece... Oct 7 11:59:31.242555564 (334h28m16.704648067s) [upload=294346]: Next returned nil segment Oct 7 11:59:31.242579049 (334h28m16.704671542s) [upload=294346]: Waiting for error group managing segments... Oct 7 11:59:31.528296503 (334h28m16.990388996s) [noise=true , address=145.79.227.16:4911 , piece_id=N5SLIS4ZCBWSRPMFTMSCLDO24XG2K7TGYPFMP3OWWI6GFXG2OIBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533220550 (334h28m16.995313033s) [noise=true , address=145.79.227.17:4969 , piece_id=VKFL26JQUWIXM63TRQUNJBR6VSNFM4L5DK52KCDYSLJG3DUO7ANQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.533794082 (334h28m16.995886565s) [noise=true , address=145.79.227.19:4077 , piece_id=FGXWAV2BLJAXOFEPPYRM6P2IKTFJ3UGZBQEIXSBHPCB4A6VWWRLA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.552365045 (334h28m17.014457528s) [noise=true , address=145.79.227.2:4114 , piece_id=DGILB62DFNR645OPZ6OCQ5EUHU4FL7ZIKGYRUV634JZNAYPRRBBQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.558314082 (334h28m17.020406565s) [noise=true , address=145.79.227.15:4873 , piece_id=OGZPXIOPPNCRKNKIDLOYOXD4HGKR5UHO732RCIZBAH3YBFTLSJSA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.592338703 (334h28m17.054431176s) [noise=true , address=145.79.227.14:4839 , piece_id=6HPWJSD6UVLEMYFTDVEWQCNWQPBKZMTHHISFM3636JCKSA5YXP3A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.618205408 (334h28m17.080297881s) [noise=true , address=77.90.31.16:4734 , piece_id=RAOY56W4WSPXWLBY26XDDVPTQQL2XAEUWDXPZUF3YZYMOP63KOPA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.635854144 (334h28m17.097946617s) [noise=true , address=45.91.116.12:4227 , piece_id=PGILI7SRQLBXXTR56B2FZDG3YXJEUGX6LDIZVWWZURSDERYAPF7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.658527770 (334h28m17.120620253s) [noise=true , address=45.91.116.4:4937 , piece_id=3OYSGEO6AJ6BP7B3OD4UK7IRLU5DWZTQME5MKYOWG32VMDTBIO3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.663561796 (334h28m17.125654279s) [noise=true , address=45.91.116.8:4045 , piece_id=2TQW73LVQVLHOORLTYIYETWEBEG3IJOBOCNQBT6GIIYFDSKOHR7A, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.669953276 (334h28m17.13204576s) [noise=true , address=45.91.116.9:4263 , piece_id=DI52AZURWWTZ7Y7G25KTUBPTWLWTRZ2QQLDYAQV6ELY4E7EPZE3Q, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.677953901 (334h28m17.140046394s) [noise=true , address=45.91.116.10:4082 , piece_id=GJY5ML63PB7R67XY73QPBL7MTBS7G3IHBTJW75NGCXW6NKELZSEA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690232242 (334h28m17.152324725s) [noise=true , address=45.91.116.14:4189 , piece_id=7YAV5ASMNRV4YMXP25JRF7FBHBU2RPNWHJDITPPYQNXCF7GHMBUA, seg_pos={0 0}, upload=294346]: Done uploading piece. err: <nil> Oct 7 11:59:31.690273119 (334h28m17.152365592s) [seg_pos={0 0}, upload=294346]: Segment reached optimal threshold of 13 pieces. Oct 7 11:59:31.690445387 (334h28m17.15253787s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690457711 (334h28m17.152550184s) [noise=true , address=45.91.116.13:4153 , piece_id=PHEHVP6DKIIYYKSEL3IWGTD66X2UOZWHRJMGQGB4JQHI6SRRVVIQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690556630 (334h28m17.152649103s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690563843 (334h28m17.152656317s) [noise=true , address=77.90.31.11:4554 , piece_id=HY3VXBJT347G3H5XR2GHOLUHPLBFLG645TZZHAFVFBDN5PPTWSTQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690644066 (334h28m17.152736539s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 11:59:31.690675746 (334h28m17.152768229s) [noise=true , address=45.91.116.6:4974 , piece_id=CCEBV33DZI5M7AS77WBJKAP4A5D7KSVZIBK5QEQKWTOL5HBE4TCQ, seg_pos={0 0}, upload=294346]: Long tail cancellation due to optimal threshold Oct 7 11:59:31.690709982 (334h28m17.152802465s) [seg_pos={0 0}, upload=294346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 11:59:31.690833988 (334h28m17.152926471s) [upload=294346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 11:59:31.690841282 (334h28m17.152933755s) [upload=294346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 11:59:32.150636356 (334h28m17.612728829s) [upload=294346]: Done uploading segments... Oct 7 11:59:32.150660100 (334h28m17.612752573s) [upload=294346]: Upload finished. err: <nil> Duration: 30.002370786s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.126093809s FAIL (2025-10-07 11:22:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.126126341s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.254174189s FAIL (2025-10-07 11:22:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.254199147s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)