Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.665892434s FAIL (2025-09-07 14:03:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.665914224s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.XX.....] duration: 22.986302472s FAIL (2025-09-07 14:03:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:59:20.726929763 (24h30m56.766502873s) [upload=21604]: Starting upload Sep 7 13:59:20.727097454 (24h30m56.766670584s) [upload=21604]: Done starting upload Sep 7 13:59:20.727226641 (24h30m56.766799761s) [upload=21604]: Uploading segments... Sep 7 13:59:20.727325441 (24h30m56.766898561s) [upload=21604]: Got next segment. Inline: false Sep 7 13:59:20.727366829 (24h30m56.766939949s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:59:20.727371478 (24h30m56.766944588s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:59:21.206412957 (24h30m57.245986077s) [seg_pos={0 0}, upload=21604]: Begin upload segment... Sep 7 13:59:21.206453635 (24h30m57.246026755s) [seg_pos={0 0}, upload=21604]: Done begin upload segment. Sep 7 13:59:21.206476920 (24h30m57.24605003s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206498099 (24h30m57.24607122s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206835486 (24h30m57.246408606s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206844062 (24h30m57.246417182s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207111774 (24h30m57.246684894s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207203520 (24h30m57.24677664s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207378084 (24h30m57.246951194s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207477985 (24h30m57.247051105s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207646006 (24h30m57.247219116s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207825730 (24h30m57.24739884s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207880315 (24h30m57.247453435s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208083483 (24h30m57.247656603s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208123550 (24h30m57.24769667s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208347258 (24h30m57.247920378s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208366606 (24h30m57.247939726s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208927508 (24h30m57.248500628s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:23.069992606 (24h30m59.109565726s) [upload=21604]: Next returned nil segment Sep 7 13:59:23.070013115 (24h30m59.109586235s) [upload=21604]: Waiting for error group managing segments... Sep 7 13:59:23.439701644 (24h30m59.479274764s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.443246758 (24h30m59.482819878s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.449626145 (24h30m59.489199255s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.463825436 (24h30m59.503398556s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.531633859 (24h30m59.571206979s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.535798629 (24h30m59.575371749s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.568030104 (24h30m59.607603224s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.580973452 (24h30m59.620546572s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.741313697 (24h30m59.780886807s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.766668605 (24h30m59.806241715s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.853782620 (24h30m59.89335574s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.036867868 (24h31m0.076440988s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161400835 (24h31m0.200973955s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161450981 (24h31m0.201024101s) [seg_pos={0 0}, upload=21604]: Segment reached optimal threshold of 13 pieces. Sep 7 13:59:24.161565981 (24h31m0.201139101s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161670982 (24h31m0.201244102s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161701230 (24h31m0.201286804s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161734825 (24h31m0.201307945s) [seg_pos={0 0}, upload=21604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:59:24.161852690 (24h31m0.20142581s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:59:24.161858060 (24h31m0.20143117s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:59:24.613922758 (24h31m0.653495878s) [upload=21604]: Done uploading segments... Sep 7 13:59:24.613939790 (24h31m0.65351291s) [upload=21604]: Upload finished. err: <nil> Duration: 22.986327681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.XX....] duration: 20.380174178s FAIL (2025-09-07 14:02:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:59:20.726929763 (24h30m56.766502873s) [upload=21604]: Starting upload Sep 7 13:59:20.727097454 (24h30m56.766670584s) [upload=21604]: Done starting upload Sep 7 13:59:20.727226641 (24h30m56.766799761s) [upload=21604]: Uploading segments... Sep 7 13:59:20.727325441 (24h30m56.766898561s) [upload=21604]: Got next segment. Inline: false Sep 7 13:59:20.727366829 (24h30m56.766939949s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:59:20.727371478 (24h30m56.766944588s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:59:21.206412957 (24h30m57.245986077s) [seg_pos={0 0}, upload=21604]: Begin upload segment... Sep 7 13:59:21.206453635 (24h30m57.246026755s) [seg_pos={0 0}, upload=21604]: Done begin upload segment. Sep 7 13:59:21.206476920 (24h30m57.24605003s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206498099 (24h30m57.24607122s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206835486 (24h30m57.246408606s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206844062 (24h30m57.246417182s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207111774 (24h30m57.246684894s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207203520 (24h30m57.24677664s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207378084 (24h30m57.246951194s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207477985 (24h30m57.247051105s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207646006 (24h30m57.247219116s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207825730 (24h30m57.24739884s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207880315 (24h30m57.247453435s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208083483 (24h30m57.247656603s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208123550 (24h30m57.24769667s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208347258 (24h30m57.247920378s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208366606 (24h30m57.247939726s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208927508 (24h30m57.248500628s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:23.069992606 (24h30m59.109565726s) [upload=21604]: Next returned nil segment Sep 7 13:59:23.070013115 (24h30m59.109586235s) [upload=21604]: Waiting for error group managing segments... Sep 7 13:59:23.439701644 (24h30m59.479274764s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.443246758 (24h30m59.482819878s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.449626145 (24h30m59.489199255s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.463825436 (24h30m59.503398556s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.531633859 (24h30m59.571206979s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.535798629 (24h30m59.575371749s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.568030104 (24h30m59.607603224s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.580973452 (24h30m59.620546572s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.741313697 (24h30m59.780886807s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.766668605 (24h30m59.806241715s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.853782620 (24h30m59.89335574s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.036867868 (24h31m0.076440988s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161400835 (24h31m0.200973955s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161450981 (24h31m0.201024101s) [seg_pos={0 0}, upload=21604]: Segment reached optimal threshold of 13 pieces. Sep 7 13:59:24.161565981 (24h31m0.201139101s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161670982 (24h31m0.201244102s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161701230 (24h31m0.201286804s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161734825 (24h31m0.201307945s) [seg_pos={0 0}, upload=21604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:59:24.161852690 (24h31m0.20142581s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:59:24.161858060 (24h31m0.20143117s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:59:24.613922758 (24h31m0.653495878s) [upload=21604]: Done uploading segments... Sep 7 13:59:24.613939790 (24h31m0.65351291s) [upload=21604]: Upload finished. err: <nil> Duration: 20.380198804s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.270780275s FAIL (2025-09-07 14:02:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.270804852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.XX...] duration: 26.985742689s FAIL (2025-09-07 14:02:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:59:20.726929763 (24h30m56.766502873s) [upload=21604]: Starting upload Sep 7 13:59:20.727097454 (24h30m56.766670584s) [upload=21604]: Done starting upload Sep 7 13:59:20.727226641 (24h30m56.766799761s) [upload=21604]: Uploading segments... Sep 7 13:59:20.727325441 (24h30m56.766898561s) [upload=21604]: Got next segment. Inline: false Sep 7 13:59:20.727366829 (24h30m56.766939949s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:59:20.727371478 (24h30m56.766944588s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:59:21.206412957 (24h30m57.245986077s) [seg_pos={0 0}, upload=21604]: Begin upload segment... Sep 7 13:59:21.206453635 (24h30m57.246026755s) [seg_pos={0 0}, upload=21604]: Done begin upload segment. Sep 7 13:59:21.206476920 (24h30m57.24605003s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206498099 (24h30m57.24607122s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206835486 (24h30m57.246408606s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206844062 (24h30m57.246417182s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207111774 (24h30m57.246684894s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207203520 (24h30m57.24677664s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207378084 (24h30m57.246951194s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207477985 (24h30m57.247051105s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207646006 (24h30m57.247219116s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207825730 (24h30m57.24739884s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207880315 (24h30m57.247453435s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208083483 (24h30m57.247656603s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208123550 (24h30m57.24769667s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208347258 (24h30m57.247920378s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208366606 (24h30m57.247939726s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208927508 (24h30m57.248500628s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:23.069992606 (24h30m59.109565726s) [upload=21604]: Next returned nil segment Sep 7 13:59:23.070013115 (24h30m59.109586235s) [upload=21604]: Waiting for error group managing segments... Sep 7 13:59:23.439701644 (24h30m59.479274764s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.443246758 (24h30m59.482819878s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.449626145 (24h30m59.489199255s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.463825436 (24h30m59.503398556s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.531633859 (24h30m59.571206979s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.535798629 (24h30m59.575371749s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.568030104 (24h30m59.607603224s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.580973452 (24h30m59.620546572s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.741313697 (24h30m59.780886807s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.766668605 (24h30m59.806241715s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.853782620 (24h30m59.89335574s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.036867868 (24h31m0.076440988s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161400835 (24h31m0.200973955s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161450981 (24h31m0.201024101s) [seg_pos={0 0}, upload=21604]: Segment reached optimal threshold of 13 pieces. Sep 7 13:59:24.161565981 (24h31m0.201139101s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161670982 (24h31m0.201244102s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161701230 (24h31m0.201286804s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161734825 (24h31m0.201307945s) [seg_pos={0 0}, upload=21604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:59:24.161852690 (24h31m0.20142581s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:59:24.161858060 (24h31m0.20143117s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:59:24.613922758 (24h31m0.653495878s) [upload=21604]: Done uploading segments... Sep 7 13:59:24.613939790 (24h31m0.65351291s) [upload=21604]: Upload finished. err: <nil> Duration: 26.985771545s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.949125909s FAIL (2025-09-07 14:01:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.94915241s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.XX..] duration: 16.161345947s FAIL (2025-09-07 14:01:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:59:20.726929763 (24h30m56.766502873s) [upload=21604]: Starting upload Sep 7 13:59:20.727097454 (24h30m56.766670584s) [upload=21604]: Done starting upload Sep 7 13:59:20.727226641 (24h30m56.766799761s) [upload=21604]: Uploading segments... Sep 7 13:59:20.727325441 (24h30m56.766898561s) [upload=21604]: Got next segment. Inline: false Sep 7 13:59:20.727366829 (24h30m56.766939949s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:59:20.727371478 (24h30m56.766944588s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:59:21.206412957 (24h30m57.245986077s) [seg_pos={0 0}, upload=21604]: Begin upload segment... Sep 7 13:59:21.206453635 (24h30m57.246026755s) [seg_pos={0 0}, upload=21604]: Done begin upload segment. Sep 7 13:59:21.206476920 (24h30m57.24605003s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206498099 (24h30m57.24607122s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206835486 (24h30m57.246408606s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206844062 (24h30m57.246417182s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207111774 (24h30m57.246684894s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207203520 (24h30m57.24677664s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207378084 (24h30m57.246951194s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207477985 (24h30m57.247051105s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207646006 (24h30m57.247219116s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207825730 (24h30m57.24739884s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207880315 (24h30m57.247453435s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208083483 (24h30m57.247656603s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208123550 (24h30m57.24769667s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208347258 (24h30m57.247920378s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208366606 (24h30m57.247939726s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208927508 (24h30m57.248500628s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:23.069992606 (24h30m59.109565726s) [upload=21604]: Next returned nil segment Sep 7 13:59:23.070013115 (24h30m59.109586235s) [upload=21604]: Waiting for error group managing segments... Sep 7 13:59:23.439701644 (24h30m59.479274764s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.443246758 (24h30m59.482819878s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.449626145 (24h30m59.489199255s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.463825436 (24h30m59.503398556s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.531633859 (24h30m59.571206979s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.535798629 (24h30m59.575371749s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.568030104 (24h30m59.607603224s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.580973452 (24h30m59.620546572s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.741313697 (24h30m59.780886807s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.766668605 (24h30m59.806241715s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.853782620 (24h30m59.89335574s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.036867868 (24h31m0.076440988s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161400835 (24h31m0.200973955s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161450981 (24h31m0.201024101s) [seg_pos={0 0}, upload=21604]: Segment reached optimal threshold of 13 pieces. Sep 7 13:59:24.161565981 (24h31m0.201139101s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161670982 (24h31m0.201244102s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161701230 (24h31m0.201286804s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161734825 (24h31m0.201307945s) [seg_pos={0 0}, upload=21604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:59:24.161852690 (24h31m0.20142581s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:59:24.161858060 (24h31m0.20143117s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:59:24.613922758 (24h31m0.653495878s) [upload=21604]: Done uploading segments... Sep 7 13:59:24.613939790 (24h31m0.65351291s) [upload=21604]: Upload finished. err: <nil> Duration: 16.16137448s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.815522544s FAIL (2025-09-07 14:00:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.815548124s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.XX.] duration: 29.975959226s FAIL (2025-09-07 14:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:59:20.726929763 (24h30m56.766502873s) [upload=21604]: Starting upload Sep 7 13:59:20.727097454 (24h30m56.766670584s) [upload=21604]: Done starting upload Sep 7 13:59:20.727226641 (24h30m56.766799761s) [upload=21604]: Uploading segments... Sep 7 13:59:20.727325441 (24h30m56.766898561s) [upload=21604]: Got next segment. Inline: false Sep 7 13:59:20.727366829 (24h30m56.766939949s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:59:20.727371478 (24h30m56.766944588s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:59:21.206412957 (24h30m57.245986077s) [seg_pos={0 0}, upload=21604]: Begin upload segment... Sep 7 13:59:21.206453635 (24h30m57.246026755s) [seg_pos={0 0}, upload=21604]: Done begin upload segment. Sep 7 13:59:21.206476920 (24h30m57.24605003s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206498099 (24h30m57.24607122s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206835486 (24h30m57.246408606s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206844062 (24h30m57.246417182s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207111774 (24h30m57.246684894s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207203520 (24h30m57.24677664s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207378084 (24h30m57.246951194s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207477985 (24h30m57.247051105s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207646006 (24h30m57.247219116s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207825730 (24h30m57.24739884s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207880315 (24h30m57.247453435s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208083483 (24h30m57.247656603s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208123550 (24h30m57.24769667s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208347258 (24h30m57.247920378s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208366606 (24h30m57.247939726s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208927508 (24h30m57.248500628s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:23.069992606 (24h30m59.109565726s) [upload=21604]: Next returned nil segment Sep 7 13:59:23.070013115 (24h30m59.109586235s) [upload=21604]: Waiting for error group managing segments... Sep 7 13:59:23.439701644 (24h30m59.479274764s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.443246758 (24h30m59.482819878s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.449626145 (24h30m59.489199255s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.463825436 (24h30m59.503398556s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.531633859 (24h30m59.571206979s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.535798629 (24h30m59.575371749s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.568030104 (24h30m59.607603224s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.580973452 (24h30m59.620546572s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.741313697 (24h30m59.780886807s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.766668605 (24h30m59.806241715s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.853782620 (24h30m59.89335574s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.036867868 (24h31m0.076440988s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161400835 (24h31m0.200973955s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161450981 (24h31m0.201024101s) [seg_pos={0 0}, upload=21604]: Segment reached optimal threshold of 13 pieces. Sep 7 13:59:24.161565981 (24h31m0.201139101s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161670982 (24h31m0.201244102s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161701230 (24h31m0.201286804s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161734825 (24h31m0.201307945s) [seg_pos={0 0}, upload=21604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:59:24.161852690 (24h31m0.20142581s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:59:24.161858060 (24h31m0.20143117s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:59:24.613922758 (24h31m0.653495878s) [upload=21604]: Done uploading segments... Sep 7 13:59:24.613939790 (24h31m0.65351291s) [upload=21604]: Upload finished. err: <nil> Duration: 29.976048498s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.548367286s FAIL (2025-09-07 14:00:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.548397623s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.XX] duration: 30.001488829s FAIL (2025-09-07 13:59:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:59:20.726929763 (24h30m56.766502873s) [upload=21604]: Starting upload Sep 7 13:59:20.727097454 (24h30m56.766670584s) [upload=21604]: Done starting upload Sep 7 13:59:20.727226641 (24h30m56.766799761s) [upload=21604]: Uploading segments... Sep 7 13:59:20.727325441 (24h30m56.766898561s) [upload=21604]: Got next segment. Inline: false Sep 7 13:59:20.727366829 (24h30m56.766939949s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:59:20.727371478 (24h30m56.766944588s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:59:21.206412957 (24h30m57.245986077s) [seg_pos={0 0}, upload=21604]: Begin upload segment... Sep 7 13:59:21.206453635 (24h30m57.246026755s) [seg_pos={0 0}, upload=21604]: Done begin upload segment. Sep 7 13:59:21.206476920 (24h30m57.24605003s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206498099 (24h30m57.24607122s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206835486 (24h30m57.246408606s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.206844062 (24h30m57.246417182s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207111774 (24h30m57.246684894s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207203520 (24h30m57.24677664s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207378084 (24h30m57.246951194s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207477985 (24h30m57.247051105s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207646006 (24h30m57.247219116s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207825730 (24h30m57.24739884s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.207880315 (24h30m57.247453435s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208083483 (24h30m57.247656603s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208123550 (24h30m57.24769667s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208347258 (24h30m57.247920378s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208366606 (24h30m57.247939726s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:21.208927508 (24h30m57.248500628s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Uploading piece... Sep 7 13:59:23.069992606 (24h30m59.109565726s) [upload=21604]: Next returned nil segment Sep 7 13:59:23.070013115 (24h30m59.109586235s) [upload=21604]: Waiting for error group managing segments... Sep 7 13:59:23.439701644 (24h30m59.479274764s) [noise=true , address=77.90.31.12:4592 , piece_id=7EP3P3YECCH7KPTVH5NDJBVDE65JESUQ4BZXZY6BXOKSY4B7YOGQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.443246758 (24h30m59.482819878s) [noise=true , address=45.91.116.12:4226 , piece_id=OI5JJNZATNPHBYJ3H4CRZ4BK2KZVVIQN74URSN53YEJVHNJ6D4XQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.449626145 (24h30m59.489199255s) [noise=true , address=45.91.116.9:4263 , piece_id=VIMJWL5CFQELHARRPIB23ZELPITM2ORG6SB7BNKWTLQ76PJ3P5IA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.463825436 (24h30m59.503398556s) [noise=true , address=45.91.116.5:4298 , piece_id=DFELY7CQ5P43JU5ALAUX5BWY5GXLFCFVG2UDEC6K5CZZZW6LCEEQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.531633859 (24h30m59.571206979s) [noise=true , address=45.91.116.10:4083 , piece_id=KDXZI7QWRPFQB4XERBWI53HA7GOCCRCONPJBDPHIYKDJS5ABJYNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.535798629 (24h30m59.575371749s) [noise=true , address=77.90.31.13:4628 , piece_id=JSV3KQXF3NBNVH4JV7LRO5YICZE4CCGPJIYD3AT2KI2D5RBODI7Q, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.568030104 (24h30m59.607603224s) [noise=true , address=45.91.116.14:4191 , piece_id=PKETGCNFEBTIG3MEBGWGVRMKJQPO4APWOV3DTCNOSXH2MGVXFBNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.580973452 (24h30m59.620546572s) [noise=true , address=77.90.31.21:4878 , piece_id=P642FWRT5R7TQXTTXMOSW3SN7GEKUEPV7WTIGHNVD4DMNBGDP77A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.741313697 (24h30m59.780886807s) [noise=true , address=45.91.116.10:4081 , piece_id=EQYPFEARAVSXYAX4RT72JOBMCUNDRCVN4XIHAJDSTCNQ4JPEDZFQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.766668605 (24h30m59.806241715s) [noise=true , address=77.90.31.19:4843 , piece_id=JXM2PFQMPVZIRHZG64KR6X2CJER6MS3DSSYOR7MUHSI4EZMX6J6A, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:23.853782620 (24h30m59.89335574s) [noise=true , address=45.91.116.11:4119 , piece_id=QUOITRXU3DYKF54FYFUZ6AF4KBSJR3LJYNFI7DG4ESVZ5P534GMQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.036867868 (24h31m0.076440988s) [noise=true , address=77.90.31.10:4519 , piece_id=FHFE4WQF6V3TWQY4PT7FS3WBEBEM2F7GQXSUHRDA4KHINAQHAVSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161400835 (24h31m0.200973955s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7N33U2NX3KAQRTBIUR7IEZQHHHM2BU5VIQKH4NHUTGA2O6XFJSA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: <nil> Sep 7 13:59:24.161450981 (24h31m0.201024101s) [seg_pos={0 0}, upload=21604]: Segment reached optimal threshold of 13 pieces. Sep 7 13:59:24.161565981 (24h31m0.201139101s) [noise=true , address=45.91.116.8:4045 , piece_id=K6YUDDQ755WU4ZHZ4XBPZMUNN7GAU3O3U5MHOU4KCF5FQT63A2PQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161670982 (24h31m0.201244102s) [noise=true , address=77.90.31.16:4734 , piece_id=6M7DI4E3MGN2ABRQ34HFUJ54H3PRMQJRDYHH4NKPDIHHTAVVWZNQ, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161701230 (24h31m0.201286804s) [noise=true , address=45.91.116.4:4937 , piece_id=6YJPZ7F2VZY4ZQM2K7MTFURQJRNTVY2O3ZPREKFIQ64DU43LXNXA, seg_pos={0 0}, upload=21604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Sep 7 13:59:24.161734825 (24h31m0.201307945s) [seg_pos={0 0}, upload=21604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:59:24.161852690 (24h31m0.20142581s) [upload=21604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:59:24.161858060 (24h31m0.20143117s) [upload=21604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:59:24.613922758 (24h31m0.653495878s) [upload=21604]: Done uploading segments... Sep 7 13:59:24.613939790 (24h31m0.65351291s) [upload=21604]: Upload finished. err: <nil> Duration: 30.001503828s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.42140494s FAIL (2025-09-07 13:59:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.421424727s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.X] duration: 30.00316756s FAIL (2025-09-07 13:58:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:58:14.473022936 (24h29m50.512596056s) [upload=21586]: Starting upload Sep 7 13:58:14.473080446 (24h29m50.512653556s) [upload=21586]: Done starting upload Sep 7 13:58:14.473219663 (24h29m50.512792823s) [upload=21586]: Uploading segments... Sep 7 13:58:14.473321969 (24h29m50.512895089s) [upload=21586]: Got next segment. Inline: false Sep 7 13:58:14.473369018 (24h29m50.512942138s) [upload=21586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:58:14.473375481 (24h29m50.512948601s) [upload=21586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:58:17.416406248 (24h29m53.455979378s) [seg_pos={0 0}, upload=21586]: Begin upload segment... Sep 7 13:58:17.416469679 (24h29m53.456042799s) [seg_pos={0 0}, upload=21586]: Done begin upload segment. Sep 7 13:58:17.416502403 (24h29m53.456075523s) [noise=true , address=45.91.116.5:4298 , piece_id=DKJDBJ74ZF3TPY2PX5CWAF3HBP3GQ5KE4DUDJZY5VYMZUAYQFO6Q, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.416517591 (24h29m53.456090731s) [noise=true , address=45.91.116.8:4045 , piece_id=LQ6ZBQC6ER4DSHVNS7NAP5A4RB5EJOOSYNDXG7P5RGBT67BSFD6A, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.417345706 (24h29m53.456918826s) [noise=true , address=45.91.116.6:4973 , piece_id=XE5JK3YBEXIYKYHD522QUH72PSDCHW3FC6K4YHSOB62H7PPM2CCQ, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.417361737 (24h29m53.456934877s) [noise=true , address=77.90.31.19:4843 , piece_id=IG5SUNXAF3R5IZ4OHZQRGTWGVPP2HN73G5Z3BBF3KBEJJDBSR3CA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.418035246 (24h29m53.457608366s) [noise=true , address=77.90.31.16:4733 , piece_id=QZSZS4DH6MJ25EVBPDTY5KA526AVAI6ZRRIO57H6QDADRTBRVHXQ, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.418042429 (24h29m53.457615559s) [noise=true , address=77.90.31.12:4590 , piece_id=N6J5FFTN6UUISD5ZLFL2SYZGVKWA63MJD5MXPOII2MYWAAWTPTTA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.418511807 (24h29m53.458084937s) [noise=true , address=77.90.31.18:4807 , piece_id=4ODHK7JIK5ZB54HOWHSZY7MX52YNGXOINABQMGCA5AVSSPBNKD5A, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.418574247 (24h29m53.458147367s) [noise=true , address=45.91.116.10:4081 , piece_id=6IS6EJTFXTBF7LIRCBEB7GWY3VVY36WG5ACUB3G227DZQGBRWDGA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.418880503 (24h29m53.458453633s) [noise=true , address=77.90.31.19:4842 , piece_id=IO47WPXICRZQVGA32E5IDEFKFKZ5TMHLRP33OLNEKRKTJDGTZX7A, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.419028556 (24h29m53.458601686s) [noise=true , address=45.91.116.14:4191 , piece_id=PA2TMKBUPAXRUSGX26O6UJ6XZAW6O3Z5HMZQRRUKWUNHTEQMSDHA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.419197509 (24h29m53.458770629s) [noise=true , address=77.90.31.15:4697 , piece_id=2BCYUVCFDJT2TYRF56QAN35KODGED2Y7E3NCWMRG5SKGCL55G3FA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.419502713 (24h29m53.459075843s) [noise=true , address=45.91.116.6:4974 , piece_id=N25CTO7BYUKPEVXTZBPKE5QJW6ITU2N6H4DS5QZVXT4XJJZGVRKA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.419800543 (24h29m53.459373663s) [noise=true , address=77.90.31.17:4772 , piece_id=4MLXMR5UXFW4VZLEDC5TFE3DK2PCI4CG6BZEZNC3ESQ6V5E6MUEA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.420120335 (24h29m53.459693466s) [noise=true , address=77.90.31.14:4663 , piece_id=QU22GKDAQHIFDL6YDLZP63AZ7WJ2EGZWYVH7QUSD5NSTBJJS5RMQ, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.420609151 (24h29m53.460182281s) [noise=true , address=45.91.116.7:4009 , piece_id=JE56Q26L645MZWRTXFHJZU5IBWSZGMHSMGDJT53IVCRMXFUH3SRA, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:17.420684956 (24h29m53.460258076s) [noise=true , address=77.90.31.10:4519 , piece_id=BG4BGAHOLKF267MEAFCR2PCC3SK6RHDDVYM6MPGIEAEGTRHV24UQ, seg_pos={0 0}, upload=21586]: Uploading piece... Sep 7 13:58:19.252411654 (24h29m55.291984774s) [upload=21586]: Next returned nil segment Sep 7 13:58:19.252430791 (24h29m55.292003911s) [upload=21586]: Waiting for error group managing segments... Sep 7 13:58:19.607663848 (24h29m55.647237009s) [noise=true , address=45.91.116.5:4298 , piece_id=DKJDBJ74ZF3TPY2PX5CWAF3HBP3GQ5KE4DUDJZY5VYMZUAYQFO6Q, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.617823640 (24h29m55.65739676s) [noise=true , address=77.90.31.12:4590 , piece_id=N6J5FFTN6UUISD5ZLFL2SYZGVKWA63MJD5MXPOII2MYWAAWTPTTA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.635623269 (24h29m55.675196389s) [noise=true , address=45.91.116.6:4973 , piece_id=XE5JK3YBEXIYKYHD522QUH72PSDCHW3FC6K4YHSOB62H7PPM2CCQ, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.635676992 (24h29m55.675250112s) [noise=true , address=77.90.31.19:4842 , piece_id=IO47WPXICRZQVGA32E5IDEFKFKZ5TMHLRP33OLNEKRKTJDGTZX7A, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.644246610 (24h29m55.68381973s) [noise=true , address=45.91.116.14:4191 , piece_id=PA2TMKBUPAXRUSGX26O6UJ6XZAW6O3Z5HMZQRRUKWUNHTEQMSDHA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.657455205 (24h29m55.697028315s) [noise=true , address=77.90.31.14:4663 , piece_id=QU22GKDAQHIFDL6YDLZP63AZ7WJ2EGZWYVH7QUSD5NSTBJJS5RMQ, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.660313604 (24h29m55.699886724s) [noise=true , address=45.91.116.10:4081 , piece_id=6IS6EJTFXTBF7LIRCBEB7GWY3VVY36WG5ACUB3G227DZQGBRWDGA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.676132134 (24h29m55.715705314s) [noise=true , address=77.90.31.16:4733 , piece_id=QZSZS4DH6MJ25EVBPDTY5KA526AVAI6ZRRIO57H6QDADRTBRVHXQ, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.678388973 (24h29m55.717962093s) [noise=true , address=77.90.31.15:4697 , piece_id=2BCYUVCFDJT2TYRF56QAN35KODGED2Y7E3NCWMRG5SKGCL55G3FA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.705503697 (24h29m55.745076817s) [noise=true , address=77.90.31.19:4843 , piece_id=IG5SUNXAF3R5IZ4OHZQRGTWGVPP2HN73G5Z3BBF3KBEJJDBSR3CA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.738409042 (24h29m55.777982162s) [noise=true , address=45.91.116.6:4974 , piece_id=N25CTO7BYUKPEVXTZBPKE5QJW6ITU2N6H4DS5QZVXT4XJJZGVRKA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.754042238 (24h29m55.793615358s) [noise=true , address=45.91.116.8:4045 , piece_id=LQ6ZBQC6ER4DSHVNS7NAP5A4RB5EJOOSYNDXG7P5RGBT67BSFD6A, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.765106418 (24h29m55.804679538s) [noise=true , address=77.90.31.18:4807 , piece_id=4ODHK7JIK5ZB54HOWHSZY7MX52YNGXOINABQMGCA5AVSSPBNKD5A, seg_pos={0 0}, upload=21586]: Done uploading piece. err: <nil> Sep 7 13:58:19.765142608 (24h29m55.804715788s) [seg_pos={0 0}, upload=21586]: Segment reached optimal threshold of 13 pieces. Sep 7 13:58:19.765387276 (24h29m55.804960397s) [noise=true , address=77.90.31.17:4772 , piece_id=4MLXMR5UXFW4VZLEDC5TFE3DK2PCI4CG6BZEZNC3ESQ6V5E6MUEA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Sep 7 13:58:19.765444305 (24h29m55.805017426s) [noise=true , address=45.91.116.7:4009 , piece_id=JE56Q26L645MZWRTXFHJZU5IBWSZGMHSMGDJT53IVCRMXFUH3SRA, seg_pos={0 0}, upload=21586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:41272->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:41272->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41272->45.91.116.7:4009: use of closed network connection Sep 7 13:58:19.765466318 (24h29m55.805039438s) [noise=true , address=77.90.31.10:4519 , piece_id=BG4BGAHOLKF267MEAFCR2PCC3SK6RHDDVYM6MPGIEAEGTRHV24UQ, seg_pos={0 0}, upload=21586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:60412->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:60412->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60412->77.90.31.10:4519: use of closed network connection Sep 7 13:58:19.765482508 (24h29m55.805055628s) [seg_pos={0 0}, upload=21586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:58:19.765552843 (24h29m55.805125953s) [upload=21586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:58:19.765557051 (24h29m55.805130171s) [upload=21586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:58:20.116358028 (24h29m56.155931148s) [upload=21586]: Done uploading segments... Sep 7 13:58:20.116381323 (24h29m56.155954443s) [upload=21586]: Upload finished. err: <nil> Duration: 30.003184392s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.105450499s FAIL (2025-09-07 13:58:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.105490295s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X.] duration: 21.255934247s FAIL (2025-09-07 13:58:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:56:34.879214136 (24h28m10.918787256s) [upload=21561]: Starting upload Sep 7 13:56:34.879273139 (24h28m10.918846259s) [upload=21561]: Done starting upload Sep 7 13:56:34.879393909 (24h28m10.918967029s) [upload=21561]: Uploading segments... Sep 7 13:56:34.879628398 (24h28m10.919201508s) [upload=21561]: Got next segment. Inline: false Sep 7 13:56:34.879656712 (24h28m10.919229833s) [upload=21561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:56:34.879662423 (24h28m10.919235553s) [upload=21561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:56:35.181788991 (24h28m11.221362111s) [seg_pos={0 0}, upload=21561]: Begin upload segment... Sep 7 13:56:35.181881137 (24h28m11.221454257s) [seg_pos={0 0}, upload=21561]: Done begin upload segment. Sep 7 13:56:35.181932966 (24h28m11.221506086s) [noise=true , address=45.91.116.7:4009 , piece_id=MAHACFMMQPXUQAITSJSWEI5D2VEYY4Z6DG7RY53IYNR3DTYFE2HA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.181957834 (24h28m11.221530954s) [noise=true , address=45.91.116.12:4226 , piece_id=7IU2MI3QNTH2FI3LRI7ZBRB262UMBASOIAYNFLXNEIXVIW53XUOQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.182643105 (24h28m11.222216225s) [noise=true , address=77.90.31.10:4519 , piece_id=EDWBO56KC43AD57ATP5ILVDA4DHLUCCFN7XUBIILS327LIKPAVWQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.182661080 (24h28m11.2222342s) [noise=true , address=45.91.116.10:4083 , piece_id=SH5TZMUJYB2S76HSHOPONUMEW36URD45QJHODWDBL4GOKCKW6V6A, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.183005679 (24h28m11.222578799s) [noise=true , address=45.91.116.5:4298 , piece_id=LMFK3MNX3GNAGFK7Y7PTVB66J67WGSLCH7FM57LNP62XVUZRQHMA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.183034423 (24h28m11.222607544s) [noise=true , address=45.91.116.14:4189 , piece_id=OQIW25XDESUI2FXU5CBKOQ7ULLK3UNLVORUQVTOKSA4OTTUOPD7A, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.183444468 (24h28m11.223017588s) [noise=true , address=45.91.116.4:4937 , piece_id=VMQVVSDNMAY37FU2WCE5XSY4CHCPNTVIUU2OBVI2JH3DGGKCQTUA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.184582216 (24h28m11.224155336s) [noise=true , address=77.90.31.15:4699 , piece_id=P3SPURTHFSW5RM5YPA67VUK5XPZJ6PR6BGLKJSCYDGC6UGEGG2TA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.184947234 (24h28m11.224520364s) [noise=true , address=45.91.116.6:4973 , piece_id=HAQXHEG3LDDFD4BVWQDJKRGSVK4ARZNP64Q375LIBODQINQY3KEQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.186631617 (24h28m11.226204737s) [noise=true , address=45.91.116.10:4081 , piece_id=6DZJS4RZENWXVTZJCP6HBKFUXKGRTWU7I32LGZUAHKDUHFANNERQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.186991085 (24h28m11.226564205s) [noise=true , address=45.91.116.10:4082 , piece_id=N5SBWEDATYCV3LLH4XOXOHN27QIOS6ARQALML3OQE7LUUHWAX6WQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.187355081 (24h28m11.226928201s) [noise=true , address=45.91.116.11:4119 , piece_id=6M5QRNRS5O56R734ADUE7NHDS2BFNTD6NXKFYPB7KDKDDPRZ5FWQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.187731351 (24h28m11.227304471s) [noise=true , address=77.90.31.18:4805 , piece_id=I5YWXNC4GCIGUBRPYADUS6MF5U3DUHGGPEONLYZWZIZWFVURRMMA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.188090058 (24h28m11.227663178s) [noise=true , address=45.91.116.4:4940 , piece_id=6OOSQO375EMC7J3SNQZ2DTGC7SEGFB3CW4WNLREWDCLTQ73CTZNQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.188508619 (24h28m11.228081739s) [noise=true , address=45.91.116.14:4191 , piece_id=NUO2T76MQYE7T2J7SJ3BRYI6JRIQOGGBVAGUHVWW4TLDVEK72POQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.188863638 (24h28m11.228436769s) [noise=true , address=45.91.116.13:4153 , piece_id=IPGXEUHJBZ4KETJT5H2SL5DTJQYJYZVRA23WBSP3IMMPCURPYB5Q, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:37.053358530 (24h28m13.09293165s) [upload=21561]: Next returned nil segment Sep 7 13:56:37.053374710 (24h28m13.09294782s) [upload=21561]: Waiting for error group managing segments... Sep 7 13:56:37.363724944 (24h28m13.403298064s) [noise=true , address=45.91.116.4:4937 , piece_id=VMQVVSDNMAY37FU2WCE5XSY4CHCPNTVIUU2OBVI2JH3DGGKCQTUA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.399414857 (24h28m13.438987977s) [noise=true , address=45.91.116.5:4298 , piece_id=LMFK3MNX3GNAGFK7Y7PTVB66J67WGSLCH7FM57LNP62XVUZRQHMA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.412636928 (24h28m13.452210048s) [noise=true , address=45.91.116.10:4083 , piece_id=SH5TZMUJYB2S76HSHOPONUMEW36URD45QJHODWDBL4GOKCKW6V6A, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.437702562 (24h28m13.477275682s) [noise=true , address=45.91.116.4:4940 , piece_id=6OOSQO375EMC7J3SNQZ2DTGC7SEGFB3CW4WNLREWDCLTQ73CTZNQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.439999006 (24h28m13.479572116s) [noise=true , address=45.91.116.7:4009 , piece_id=MAHACFMMQPXUQAITSJSWEI5D2VEYY4Z6DG7RY53IYNR3DTYFE2HA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.442035043 (24h28m13.481608163s) [noise=true , address=45.91.116.10:4082 , piece_id=N5SBWEDATYCV3LLH4XOXOHN27QIOS6ARQALML3OQE7LUUHWAX6WQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.445729432 (24h28m13.485302542s) [noise=true , address=77.90.31.15:4699 , piece_id=P3SPURTHFSW5RM5YPA67VUK5XPZJ6PR6BGLKJSCYDGC6UGEGG2TA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.447915174 (24h28m13.487488294s) [noise=true , address=45.91.116.14:4189 , piece_id=OQIW25XDESUI2FXU5CBKOQ7ULLK3UNLVORUQVTOKSA4OTTUOPD7A, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.452173803 (24h28m13.491746923s) [noise=true , address=45.91.116.10:4081 , piece_id=6DZJS4RZENWXVTZJCP6HBKFUXKGRTWU7I32LGZUAHKDUHFANNERQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.457755534 (24h28m13.497328654s) [noise=true , address=77.90.31.18:4805 , piece_id=I5YWXNC4GCIGUBRPYADUS6MF5U3DUHGGPEONLYZWZIZWFVURRMMA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.461694531 (24h28m13.501267651s) [noise=true , address=45.91.116.14:4191 , piece_id=NUO2T76MQYE7T2J7SJ3BRYI6JRIQOGGBVAGUHVWW4TLDVEK72POQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.486897567 (24h28m13.526470687s) [noise=true , address=45.91.116.12:4226 , piece_id=7IU2MI3QNTH2FI3LRI7ZBRB262UMBASOIAYNFLXNEIXVIW53XUOQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.496704263 (24h28m13.536277383s) [noise=true , address=45.91.116.13:4153 , piece_id=IPGXEUHJBZ4KETJT5H2SL5DTJQYJYZVRA23WBSP3IMMPCURPYB5Q, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.496728770 (24h28m13.53630189s) [seg_pos={0 0}, upload=21561]: Segment reached optimal threshold of 13 pieces. Sep 7 13:56:37.496840133 (24h28m13.536413253s) [noise=true , address=77.90.31.10:4519 , piece_id=EDWBO56KC43AD57ATP5ILVDA4DHLUCCFN7XUBIILS327LIKPAVWQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Sep 7 13:56:37.496906180 (24h28m13.53647929s) [noise=true , address=45.91.116.6:4973 , piece_id=HAQXHEG3LDDFD4BVWQDJKRGSVK4ARZNP64Q375LIBODQINQY3KEQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Sep 7 13:56:37.496953780 (24h28m13.53652689s) [noise=true , address=45.91.116.11:4119 , piece_id=6M5QRNRS5O56R734ADUE7NHDS2BFNTD6NXKFYPB7KDKDDPRZ5FWQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Sep 7 13:56:37.496987566 (24h28m13.536560686s) [seg_pos={0 0}, upload=21561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:56:37.497054513 (24h28m13.536627633s) [upload=21561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:56:37.497058701 (24h28m13.536631821s) [upload=21561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:56:37.836312392 (24h28m13.875885512s) [upload=21561]: Done uploading segments... Sep 7 13:56:37.836331058 (24h28m13.875904178s) [upload=21561]: Upload finished. err: <nil> Duration: 21.255964936s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.683909159s FAIL (2025-09-07 13:57:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.683933044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..X] duration: 30.002166508s FAIL (2025-09-07 13:57:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:56:34.879214136 (24h28m10.918787256s) [upload=21561]: Starting upload Sep 7 13:56:34.879273139 (24h28m10.918846259s) [upload=21561]: Done starting upload Sep 7 13:56:34.879393909 (24h28m10.918967029s) [upload=21561]: Uploading segments... Sep 7 13:56:34.879628398 (24h28m10.919201508s) [upload=21561]: Got next segment. Inline: false Sep 7 13:56:34.879656712 (24h28m10.919229833s) [upload=21561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:56:34.879662423 (24h28m10.919235553s) [upload=21561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:56:35.181788991 (24h28m11.221362111s) [seg_pos={0 0}, upload=21561]: Begin upload segment... Sep 7 13:56:35.181881137 (24h28m11.221454257s) [seg_pos={0 0}, upload=21561]: Done begin upload segment. Sep 7 13:56:35.181932966 (24h28m11.221506086s) [noise=true , address=45.91.116.7:4009 , piece_id=MAHACFMMQPXUQAITSJSWEI5D2VEYY4Z6DG7RY53IYNR3DTYFE2HA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.181957834 (24h28m11.221530954s) [noise=true , address=45.91.116.12:4226 , piece_id=7IU2MI3QNTH2FI3LRI7ZBRB262UMBASOIAYNFLXNEIXVIW53XUOQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.182643105 (24h28m11.222216225s) [noise=true , address=77.90.31.10:4519 , piece_id=EDWBO56KC43AD57ATP5ILVDA4DHLUCCFN7XUBIILS327LIKPAVWQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.182661080 (24h28m11.2222342s) [noise=true , address=45.91.116.10:4083 , piece_id=SH5TZMUJYB2S76HSHOPONUMEW36URD45QJHODWDBL4GOKCKW6V6A, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.183005679 (24h28m11.222578799s) [noise=true , address=45.91.116.5:4298 , piece_id=LMFK3MNX3GNAGFK7Y7PTVB66J67WGSLCH7FM57LNP62XVUZRQHMA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.183034423 (24h28m11.222607544s) [noise=true , address=45.91.116.14:4189 , piece_id=OQIW25XDESUI2FXU5CBKOQ7ULLK3UNLVORUQVTOKSA4OTTUOPD7A, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.183444468 (24h28m11.223017588s) [noise=true , address=45.91.116.4:4937 , piece_id=VMQVVSDNMAY37FU2WCE5XSY4CHCPNTVIUU2OBVI2JH3DGGKCQTUA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.184582216 (24h28m11.224155336s) [noise=true , address=77.90.31.15:4699 , piece_id=P3SPURTHFSW5RM5YPA67VUK5XPZJ6PR6BGLKJSCYDGC6UGEGG2TA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.184947234 (24h28m11.224520364s) [noise=true , address=45.91.116.6:4973 , piece_id=HAQXHEG3LDDFD4BVWQDJKRGSVK4ARZNP64Q375LIBODQINQY3KEQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.186631617 (24h28m11.226204737s) [noise=true , address=45.91.116.10:4081 , piece_id=6DZJS4RZENWXVTZJCP6HBKFUXKGRTWU7I32LGZUAHKDUHFANNERQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.186991085 (24h28m11.226564205s) [noise=true , address=45.91.116.10:4082 , piece_id=N5SBWEDATYCV3LLH4XOXOHN27QIOS6ARQALML3OQE7LUUHWAX6WQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.187355081 (24h28m11.226928201s) [noise=true , address=45.91.116.11:4119 , piece_id=6M5QRNRS5O56R734ADUE7NHDS2BFNTD6NXKFYPB7KDKDDPRZ5FWQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.187731351 (24h28m11.227304471s) [noise=true , address=77.90.31.18:4805 , piece_id=I5YWXNC4GCIGUBRPYADUS6MF5U3DUHGGPEONLYZWZIZWFVURRMMA, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.188090058 (24h28m11.227663178s) [noise=true , address=45.91.116.4:4940 , piece_id=6OOSQO375EMC7J3SNQZ2DTGC7SEGFB3CW4WNLREWDCLTQ73CTZNQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.188508619 (24h28m11.228081739s) [noise=true , address=45.91.116.14:4191 , piece_id=NUO2T76MQYE7T2J7SJ3BRYI6JRIQOGGBVAGUHVWW4TLDVEK72POQ, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:35.188863638 (24h28m11.228436769s) [noise=true , address=45.91.116.13:4153 , piece_id=IPGXEUHJBZ4KETJT5H2SL5DTJQYJYZVRA23WBSP3IMMPCURPYB5Q, seg_pos={0 0}, upload=21561]: Uploading piece... Sep 7 13:56:37.053358530 (24h28m13.09293165s) [upload=21561]: Next returned nil segment Sep 7 13:56:37.053374710 (24h28m13.09294782s) [upload=21561]: Waiting for error group managing segments... Sep 7 13:56:37.363724944 (24h28m13.403298064s) [noise=true , address=45.91.116.4:4937 , piece_id=VMQVVSDNMAY37FU2WCE5XSY4CHCPNTVIUU2OBVI2JH3DGGKCQTUA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.399414857 (24h28m13.438987977s) [noise=true , address=45.91.116.5:4298 , piece_id=LMFK3MNX3GNAGFK7Y7PTVB66J67WGSLCH7FM57LNP62XVUZRQHMA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.412636928 (24h28m13.452210048s) [noise=true , address=45.91.116.10:4083 , piece_id=SH5TZMUJYB2S76HSHOPONUMEW36URD45QJHODWDBL4GOKCKW6V6A, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.437702562 (24h28m13.477275682s) [noise=true , address=45.91.116.4:4940 , piece_id=6OOSQO375EMC7J3SNQZ2DTGC7SEGFB3CW4WNLREWDCLTQ73CTZNQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.439999006 (24h28m13.479572116s) [noise=true , address=45.91.116.7:4009 , piece_id=MAHACFMMQPXUQAITSJSWEI5D2VEYY4Z6DG7RY53IYNR3DTYFE2HA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.442035043 (24h28m13.481608163s) [noise=true , address=45.91.116.10:4082 , piece_id=N5SBWEDATYCV3LLH4XOXOHN27QIOS6ARQALML3OQE7LUUHWAX6WQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.445729432 (24h28m13.485302542s) [noise=true , address=77.90.31.15:4699 , piece_id=P3SPURTHFSW5RM5YPA67VUK5XPZJ6PR6BGLKJSCYDGC6UGEGG2TA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.447915174 (24h28m13.487488294s) [noise=true , address=45.91.116.14:4189 , piece_id=OQIW25XDESUI2FXU5CBKOQ7ULLK3UNLVORUQVTOKSA4OTTUOPD7A, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.452173803 (24h28m13.491746923s) [noise=true , address=45.91.116.10:4081 , piece_id=6DZJS4RZENWXVTZJCP6HBKFUXKGRTWU7I32LGZUAHKDUHFANNERQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.457755534 (24h28m13.497328654s) [noise=true , address=77.90.31.18:4805 , piece_id=I5YWXNC4GCIGUBRPYADUS6MF5U3DUHGGPEONLYZWZIZWFVURRMMA, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.461694531 (24h28m13.501267651s) [noise=true , address=45.91.116.14:4191 , piece_id=NUO2T76MQYE7T2J7SJ3BRYI6JRIQOGGBVAGUHVWW4TLDVEK72POQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.486897567 (24h28m13.526470687s) [noise=true , address=45.91.116.12:4226 , piece_id=7IU2MI3QNTH2FI3LRI7ZBRB262UMBASOIAYNFLXNEIXVIW53XUOQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.496704263 (24h28m13.536277383s) [noise=true , address=45.91.116.13:4153 , piece_id=IPGXEUHJBZ4KETJT5H2SL5DTJQYJYZVRA23WBSP3IMMPCURPYB5Q, seg_pos={0 0}, upload=21561]: Done uploading piece. err: <nil> Sep 7 13:56:37.496728770 (24h28m13.53630189s) [seg_pos={0 0}, upload=21561]: Segment reached optimal threshold of 13 pieces. Sep 7 13:56:37.496840133 (24h28m13.536413253s) [noise=true , address=77.90.31.10:4519 , piece_id=EDWBO56KC43AD57ATP5ILVDA4DHLUCCFN7XUBIILS327LIKPAVWQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Sep 7 13:56:37.496906180 (24h28m13.53647929s) [noise=true , address=45.91.116.6:4973 , piece_id=HAQXHEG3LDDFD4BVWQDJKRGSVK4ARZNP64Q375LIBODQINQY3KEQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Sep 7 13:56:37.496953780 (24h28m13.53652689s) [noise=true , address=45.91.116.11:4119 , piece_id=6M5QRNRS5O56R734ADUE7NHDS2BFNTD6NXKFYPB7KDKDDPRZ5FWQ, seg_pos={0 0}, upload=21561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Sep 7 13:56:37.496987566 (24h28m13.536560686s) [seg_pos={0 0}, upload=21561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:56:37.497054513 (24h28m13.536627633s) [upload=21561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:56:37.497058701 (24h28m13.536631821s) [upload=21561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:56:37.836312392 (24h28m13.875885512s) [upload=21561]: Done uploading segments... Sep 7 13:56:37.836331058 (24h28m13.875904178s) [upload=21561]: Upload finished. err: <nil> Duration: 30.002190081s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.776843331s FAIL (2025-09-07 13:56:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.776868328s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX..] duration: 15.793758067s FAIL (2025-09-07 13:56:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:54:14.024198717 (24h25m50.063771827s) [upload=21526]: Starting upload Sep 7 13:54:14.024252109 (24h25m50.063825229s) [upload=21526]: Done starting upload Sep 7 13:54:14.024298036 (24h25m50.063871156s) [upload=21526]: Uploading segments... Sep 7 13:54:14.024531293 (24h25m50.064104413s) [upload=21526]: Got next segment. Inline: false Sep 7 13:54:14.024573835 (24h25m50.064146945s) [upload=21526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:54:14.024578373 (24h25m50.064151493s) [upload=21526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:54:14.395751352 (24h25m50.435324483s) [seg_pos={0 0}, upload=21526]: Begin upload segment... Sep 7 13:54:14.395818461 (24h25m50.435391581s) [seg_pos={0 0}, upload=21526]: Done begin upload segment. Sep 7 13:54:14.395824122 (24h25m50.435397242s) [noise=true , address=77.90.31.13:4627 , piece_id=Z4EDCR2HRPPS4RXAFFDEZHCKRKFL4ABQHN4CZMGYKZG6YJW2NKEA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.395846054 (24h25m50.435419174s) [noise=true , address=77.90.31.13:4628 , piece_id=GIRSHGXYPEO5E43IOGSX4A3WEWJD4B7AUOYAH2LSJF5VEHOMEPDA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.395927220 (24h25m50.43550034s) [noise=true , address=77.90.31.12:4592 , piece_id=CAKOSDXAXI5VWJENCCIWESXPLPSEWYDXA2DWHX3B5WWJB34BTHJA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396285615 (24h25m50.435858735s) [noise=true , address=45.91.116.4:4937 , piece_id=DA5ACPLN3VDN4HPEHDWFLLB5QVWSYSGNW67QG74Z7Y6UJHRNC3MA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396640464 (24h25m50.436213584s) [noise=true , address=45.91.116.5:4298 , piece_id=SH6JH5DCRQNI4EVS3XJJGBDJDRZGRCTZBH5BGB5RFE6RK3LKQ2JA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396947061 (24h25m50.436520191s) [noise=true , address=77.90.31.15:4698 , piece_id=OMXHZLTUBYFTLBOCFB75NAGPCWWTMDFJIKT6MJ4DP4AXXYI72SIQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397254809 (24h25m50.436827939s) [noise=true , address=77.90.31.14:4662 , piece_id=XCG4JZTQ6FAXX2XJDKCBZ3YUIUSDKU3B6CAMSLU6SRQZXMLNMHAQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397558380 (24h25m50.4371315s) [noise=true , address=45.91.116.6:4975 , piece_id=QYPTGUH4F4MVLWAMXYQLDYZ3VQ7HGZQELZXKX2VENVFFLMVIG4DA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397901337 (24h25m50.437474457s) [noise=true , address=45.91.116.10:4083 , piece_id=7RVV6MCA3LSQNL3ZAVNS7G5D3XPZORFXR5MFF37S5DRTVZUVC23Q, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398284710 (24h25m50.43785783s) [noise=true , address=45.91.116.8:4047 , piece_id=7XOVSYGGKYMQW3HKZG4BIO3F53QM424E45ITI5EPYXORPWJ3WHHQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398606416 (24h25m50.438179536s) [noise=true , address=45.91.116.11:4119 , piece_id=GMBYKDD3FOA2IUETFYO7EAB3GK4OIQC7BFGXMHMO5ZUNH4CFYLKQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398915858 (24h25m50.438488978s) [noise=true , address=77.90.31.12:4589 , piece_id=SOLDGQ5L7GKCCGIH7YXONRW2STRWTB7GMST46JHO5CIJDMJREFHQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399227524 (24h25m50.438800644s) [noise=true , address=45.91.116.12:4225 , piece_id=42ALAPORSYG7GMRJP2DQA2QE4RTMFXM6HERWFJ7OTW67R5CMAVLA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399532848 (24h25m50.439105988s) [noise=true , address=45.91.116.4:4939 , piece_id=YLPPJVHYZSYOWGADH34FJPINCGVK2OVROJOJNLKM5QQC36RYLV7Q, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399842581 (24h25m50.439415701s) [noise=true , address=45.91.116.7:4011 , piece_id=MWJRC3MGEI2HTXB7M4BI2BJ6BLKLUOGU4H7N75PMXDRXEBUQIKEA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.400212709 (24h25m50.439785829s) [noise=true , address=45.91.116.10:4081 , piece_id=WPZWZFZMVWESQHZ5UOGBA4BC3TJAF72HWMIHBFLM2UATWMZ2LZ7A, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:16.270153320 (24h25m52.30972644s) [upload=21526]: Next returned nil segment Sep 7 13:54:16.270172167 (24h25m52.309745277s) [upload=21526]: Waiting for error group managing segments... Sep 7 13:54:16.614864675 (24h25m52.654437795s) [noise=true , address=45.91.116.10:4081 , piece_id=WPZWZFZMVWESQHZ5UOGBA4BC3TJAF72HWMIHBFLM2UATWMZ2LZ7A, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.618689394 (24h25m52.658262514s) [noise=true , address=45.91.116.5:4298 , piece_id=SH6JH5DCRQNI4EVS3XJJGBDJDRZGRCTZBH5BGB5RFE6RK3LKQ2JA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.629888283 (24h25m52.669461403s) [noise=true , address=45.91.116.4:4937 , piece_id=DA5ACPLN3VDN4HPEHDWFLLB5QVWSYSGNW67QG74Z7Y6UJHRNC3MA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.633772636 (24h25m52.673345756s) [noise=true , address=77.90.31.12:4592 , piece_id=CAKOSDXAXI5VWJENCCIWESXPLPSEWYDXA2DWHX3B5WWJB34BTHJA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.637705171 (24h25m52.677278291s) [noise=true , address=45.91.116.7:4011 , piece_id=MWJRC3MGEI2HTXB7M4BI2BJ6BLKLUOGU4H7N75PMXDRXEBUQIKEA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.661186964 (24h25m52.700760084s) [noise=true , address=45.91.116.11:4119 , piece_id=GMBYKDD3FOA2IUETFYO7EAB3GK4OIQC7BFGXMHMO5ZUNH4CFYLKQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.671744326 (24h25m52.711317446s) [noise=true , address=45.91.116.4:4939 , piece_id=YLPPJVHYZSYOWGADH34FJPINCGVK2OVROJOJNLKM5QQC36RYLV7Q, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.682091876 (24h25m52.721664986s) [noise=true , address=45.91.116.12:4225 , piece_id=42ALAPORSYG7GMRJP2DQA2QE4RTMFXM6HERWFJ7OTW67R5CMAVLA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.695192555 (24h25m52.734765675s) [noise=true , address=77.90.31.12:4589 , piece_id=SOLDGQ5L7GKCCGIH7YXONRW2STRWTB7GMST46JHO5CIJDMJREFHQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.704359946 (24h25m52.743933066s) [noise=true , address=45.91.116.10:4083 , piece_id=7RVV6MCA3LSQNL3ZAVNS7G5D3XPZORFXR5MFF37S5DRTVZUVC23Q, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.712230787 (24h25m52.751803907s) [noise=true , address=77.90.31.13:4628 , piece_id=GIRSHGXYPEO5E43IOGSX4A3WEWJD4B7AUOYAH2LSJF5VEHOMEPDA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.719294514 (24h25m52.758867624s) [noise=true , address=77.90.31.15:4698 , piece_id=OMXHZLTUBYFTLBOCFB75NAGPCWWTMDFJIKT6MJ4DP4AXXYI72SIQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.726957527 (24h25m52.766530657s) [noise=true , address=45.91.116.8:4047 , piece_id=7XOVSYGGKYMQW3HKZG4BIO3F53QM424E45ITI5EPYXORPWJ3WHHQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.726985581 (24h25m52.766558711s) [seg_pos={0 0}, upload=21526]: Segment reached optimal threshold of 13 pieces. Sep 7 13:54:16.727185895 (24h25m52.766759015s) [noise=true , address=77.90.31.14:4662 , piece_id=XCG4JZTQ6FAXX2XJDKCBZ3YUIUSDKU3B6CAMSLU6SRQZXMLNMHAQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection Sep 7 13:54:16.727325081 (24h25m52.766898191s) [noise=true , address=77.90.31.13:4627 , piece_id=Z4EDCR2HRPPS4RXAFFDEZHCKRKFL4ABQHN4CZMGYKZG6YJW2NKEA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:46560->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 7 13:54:16.727358625 (24h25m52.766931745s) [noise=true , address=45.91.116.6:4975 , piece_id=QYPTGUH4F4MVLWAMXYQLDYZ3VQ7HGZQELZXKX2VENVFFLMVIG4DA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection Sep 7 13:54:16.727414582 (24h25m52.766987702s) [seg_pos={0 0}, upload=21526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:54:16.727508321 (24h25m52.767081431s) [upload=21526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:54:16.727512310 (24h25m52.76708542s) [upload=21526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:54:17.098909968 (24h25m53.138483088s) [upload=21526]: Done uploading segments... Sep 7 13:54:17.098927803 (24h25m53.138500923s) [upload=21526]: Upload finished. err: <nil> Duration: 15.793786591s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.653091127s FAIL (2025-09-07 13:56:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.653115002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX.] duration: 21.478299408s FAIL (2025-09-07 13:55:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:54:14.024198717 (24h25m50.063771827s) [upload=21526]: Starting upload Sep 7 13:54:14.024252109 (24h25m50.063825229s) [upload=21526]: Done starting upload Sep 7 13:54:14.024298036 (24h25m50.063871156s) [upload=21526]: Uploading segments... Sep 7 13:54:14.024531293 (24h25m50.064104413s) [upload=21526]: Got next segment. Inline: false Sep 7 13:54:14.024573835 (24h25m50.064146945s) [upload=21526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:54:14.024578373 (24h25m50.064151493s) [upload=21526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:54:14.395751352 (24h25m50.435324483s) [seg_pos={0 0}, upload=21526]: Begin upload segment... Sep 7 13:54:14.395818461 (24h25m50.435391581s) [seg_pos={0 0}, upload=21526]: Done begin upload segment. Sep 7 13:54:14.395824122 (24h25m50.435397242s) [noise=true , address=77.90.31.13:4627 , piece_id=Z4EDCR2HRPPS4RXAFFDEZHCKRKFL4ABQHN4CZMGYKZG6YJW2NKEA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.395846054 (24h25m50.435419174s) [noise=true , address=77.90.31.13:4628 , piece_id=GIRSHGXYPEO5E43IOGSX4A3WEWJD4B7AUOYAH2LSJF5VEHOMEPDA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.395927220 (24h25m50.43550034s) [noise=true , address=77.90.31.12:4592 , piece_id=CAKOSDXAXI5VWJENCCIWESXPLPSEWYDXA2DWHX3B5WWJB34BTHJA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396285615 (24h25m50.435858735s) [noise=true , address=45.91.116.4:4937 , piece_id=DA5ACPLN3VDN4HPEHDWFLLB5QVWSYSGNW67QG74Z7Y6UJHRNC3MA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396640464 (24h25m50.436213584s) [noise=true , address=45.91.116.5:4298 , piece_id=SH6JH5DCRQNI4EVS3XJJGBDJDRZGRCTZBH5BGB5RFE6RK3LKQ2JA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396947061 (24h25m50.436520191s) [noise=true , address=77.90.31.15:4698 , piece_id=OMXHZLTUBYFTLBOCFB75NAGPCWWTMDFJIKT6MJ4DP4AXXYI72SIQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397254809 (24h25m50.436827939s) [noise=true , address=77.90.31.14:4662 , piece_id=XCG4JZTQ6FAXX2XJDKCBZ3YUIUSDKU3B6CAMSLU6SRQZXMLNMHAQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397558380 (24h25m50.4371315s) [noise=true , address=45.91.116.6:4975 , piece_id=QYPTGUH4F4MVLWAMXYQLDYZ3VQ7HGZQELZXKX2VENVFFLMVIG4DA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397901337 (24h25m50.437474457s) [noise=true , address=45.91.116.10:4083 , piece_id=7RVV6MCA3LSQNL3ZAVNS7G5D3XPZORFXR5MFF37S5DRTVZUVC23Q, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398284710 (24h25m50.43785783s) [noise=true , address=45.91.116.8:4047 , piece_id=7XOVSYGGKYMQW3HKZG4BIO3F53QM424E45ITI5EPYXORPWJ3WHHQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398606416 (24h25m50.438179536s) [noise=true , address=45.91.116.11:4119 , piece_id=GMBYKDD3FOA2IUETFYO7EAB3GK4OIQC7BFGXMHMO5ZUNH4CFYLKQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398915858 (24h25m50.438488978s) [noise=true , address=77.90.31.12:4589 , piece_id=SOLDGQ5L7GKCCGIH7YXONRW2STRWTB7GMST46JHO5CIJDMJREFHQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399227524 (24h25m50.438800644s) [noise=true , address=45.91.116.12:4225 , piece_id=42ALAPORSYG7GMRJP2DQA2QE4RTMFXM6HERWFJ7OTW67R5CMAVLA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399532848 (24h25m50.439105988s) [noise=true , address=45.91.116.4:4939 , piece_id=YLPPJVHYZSYOWGADH34FJPINCGVK2OVROJOJNLKM5QQC36RYLV7Q, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399842581 (24h25m50.439415701s) [noise=true , address=45.91.116.7:4011 , piece_id=MWJRC3MGEI2HTXB7M4BI2BJ6BLKLUOGU4H7N75PMXDRXEBUQIKEA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.400212709 (24h25m50.439785829s) [noise=true , address=45.91.116.10:4081 , piece_id=WPZWZFZMVWESQHZ5UOGBA4BC3TJAF72HWMIHBFLM2UATWMZ2LZ7A, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:16.270153320 (24h25m52.30972644s) [upload=21526]: Next returned nil segment Sep 7 13:54:16.270172167 (24h25m52.309745277s) [upload=21526]: Waiting for error group managing segments... Sep 7 13:54:16.614864675 (24h25m52.654437795s) [noise=true , address=45.91.116.10:4081 , piece_id=WPZWZFZMVWESQHZ5UOGBA4BC3TJAF72HWMIHBFLM2UATWMZ2LZ7A, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.618689394 (24h25m52.658262514s) [noise=true , address=45.91.116.5:4298 , piece_id=SH6JH5DCRQNI4EVS3XJJGBDJDRZGRCTZBH5BGB5RFE6RK3LKQ2JA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.629888283 (24h25m52.669461403s) [noise=true , address=45.91.116.4:4937 , piece_id=DA5ACPLN3VDN4HPEHDWFLLB5QVWSYSGNW67QG74Z7Y6UJHRNC3MA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.633772636 (24h25m52.673345756s) [noise=true , address=77.90.31.12:4592 , piece_id=CAKOSDXAXI5VWJENCCIWESXPLPSEWYDXA2DWHX3B5WWJB34BTHJA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.637705171 (24h25m52.677278291s) [noise=true , address=45.91.116.7:4011 , piece_id=MWJRC3MGEI2HTXB7M4BI2BJ6BLKLUOGU4H7N75PMXDRXEBUQIKEA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.661186964 (24h25m52.700760084s) [noise=true , address=45.91.116.11:4119 , piece_id=GMBYKDD3FOA2IUETFYO7EAB3GK4OIQC7BFGXMHMO5ZUNH4CFYLKQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.671744326 (24h25m52.711317446s) [noise=true , address=45.91.116.4:4939 , piece_id=YLPPJVHYZSYOWGADH34FJPINCGVK2OVROJOJNLKM5QQC36RYLV7Q, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.682091876 (24h25m52.721664986s) [noise=true , address=45.91.116.12:4225 , piece_id=42ALAPORSYG7GMRJP2DQA2QE4RTMFXM6HERWFJ7OTW67R5CMAVLA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.695192555 (24h25m52.734765675s) [noise=true , address=77.90.31.12:4589 , piece_id=SOLDGQ5L7GKCCGIH7YXONRW2STRWTB7GMST46JHO5CIJDMJREFHQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.704359946 (24h25m52.743933066s) [noise=true , address=45.91.116.10:4083 , piece_id=7RVV6MCA3LSQNL3ZAVNS7G5D3XPZORFXR5MFF37S5DRTVZUVC23Q, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.712230787 (24h25m52.751803907s) [noise=true , address=77.90.31.13:4628 , piece_id=GIRSHGXYPEO5E43IOGSX4A3WEWJD4B7AUOYAH2LSJF5VEHOMEPDA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.719294514 (24h25m52.758867624s) [noise=true , address=77.90.31.15:4698 , piece_id=OMXHZLTUBYFTLBOCFB75NAGPCWWTMDFJIKT6MJ4DP4AXXYI72SIQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.726957527 (24h25m52.766530657s) [noise=true , address=45.91.116.8:4047 , piece_id=7XOVSYGGKYMQW3HKZG4BIO3F53QM424E45ITI5EPYXORPWJ3WHHQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.726985581 (24h25m52.766558711s) [seg_pos={0 0}, upload=21526]: Segment reached optimal threshold of 13 pieces. Sep 7 13:54:16.727185895 (24h25m52.766759015s) [noise=true , address=77.90.31.14:4662 , piece_id=XCG4JZTQ6FAXX2XJDKCBZ3YUIUSDKU3B6CAMSLU6SRQZXMLNMHAQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection Sep 7 13:54:16.727325081 (24h25m52.766898191s) [noise=true , address=77.90.31.13:4627 , piece_id=Z4EDCR2HRPPS4RXAFFDEZHCKRKFL4ABQHN4CZMGYKZG6YJW2NKEA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:46560->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 7 13:54:16.727358625 (24h25m52.766931745s) [noise=true , address=45.91.116.6:4975 , piece_id=QYPTGUH4F4MVLWAMXYQLDYZ3VQ7HGZQELZXKX2VENVFFLMVIG4DA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection Sep 7 13:54:16.727414582 (24h25m52.766987702s) [seg_pos={0 0}, upload=21526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:54:16.727508321 (24h25m52.767081431s) [upload=21526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:54:16.727512310 (24h25m52.76708542s) [upload=21526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:54:17.098909968 (24h25m53.138483088s) [upload=21526]: Done uploading segments... Sep 7 13:54:17.098927803 (24h25m53.138500923s) [upload=21526]: Upload finished. err: <nil> Duration: 21.478322692s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.713898596s FAIL (2025-09-07 13:54:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.713919325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.XX] duration: 30.001325346s FAIL (2025-09-07 13:54:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:54:14.024198717 (24h25m50.063771827s) [upload=21526]: Starting upload Sep 7 13:54:14.024252109 (24h25m50.063825229s) [upload=21526]: Done starting upload Sep 7 13:54:14.024298036 (24h25m50.063871156s) [upload=21526]: Uploading segments... Sep 7 13:54:14.024531293 (24h25m50.064104413s) [upload=21526]: Got next segment. Inline: false Sep 7 13:54:14.024573835 (24h25m50.064146945s) [upload=21526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:54:14.024578373 (24h25m50.064151493s) [upload=21526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:54:14.395751352 (24h25m50.435324483s) [seg_pos={0 0}, upload=21526]: Begin upload segment... Sep 7 13:54:14.395818461 (24h25m50.435391581s) [seg_pos={0 0}, upload=21526]: Done begin upload segment. Sep 7 13:54:14.395824122 (24h25m50.435397242s) [noise=true , address=77.90.31.13:4627 , piece_id=Z4EDCR2HRPPS4RXAFFDEZHCKRKFL4ABQHN4CZMGYKZG6YJW2NKEA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.395846054 (24h25m50.435419174s) [noise=true , address=77.90.31.13:4628 , piece_id=GIRSHGXYPEO5E43IOGSX4A3WEWJD4B7AUOYAH2LSJF5VEHOMEPDA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.395927220 (24h25m50.43550034s) [noise=true , address=77.90.31.12:4592 , piece_id=CAKOSDXAXI5VWJENCCIWESXPLPSEWYDXA2DWHX3B5WWJB34BTHJA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396285615 (24h25m50.435858735s) [noise=true , address=45.91.116.4:4937 , piece_id=DA5ACPLN3VDN4HPEHDWFLLB5QVWSYSGNW67QG74Z7Y6UJHRNC3MA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396640464 (24h25m50.436213584s) [noise=true , address=45.91.116.5:4298 , piece_id=SH6JH5DCRQNI4EVS3XJJGBDJDRZGRCTZBH5BGB5RFE6RK3LKQ2JA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.396947061 (24h25m50.436520191s) [noise=true , address=77.90.31.15:4698 , piece_id=OMXHZLTUBYFTLBOCFB75NAGPCWWTMDFJIKT6MJ4DP4AXXYI72SIQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397254809 (24h25m50.436827939s) [noise=true , address=77.90.31.14:4662 , piece_id=XCG4JZTQ6FAXX2XJDKCBZ3YUIUSDKU3B6CAMSLU6SRQZXMLNMHAQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397558380 (24h25m50.4371315s) [noise=true , address=45.91.116.6:4975 , piece_id=QYPTGUH4F4MVLWAMXYQLDYZ3VQ7HGZQELZXKX2VENVFFLMVIG4DA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.397901337 (24h25m50.437474457s) [noise=true , address=45.91.116.10:4083 , piece_id=7RVV6MCA3LSQNL3ZAVNS7G5D3XPZORFXR5MFF37S5DRTVZUVC23Q, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398284710 (24h25m50.43785783s) [noise=true , address=45.91.116.8:4047 , piece_id=7XOVSYGGKYMQW3HKZG4BIO3F53QM424E45ITI5EPYXORPWJ3WHHQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398606416 (24h25m50.438179536s) [noise=true , address=45.91.116.11:4119 , piece_id=GMBYKDD3FOA2IUETFYO7EAB3GK4OIQC7BFGXMHMO5ZUNH4CFYLKQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.398915858 (24h25m50.438488978s) [noise=true , address=77.90.31.12:4589 , piece_id=SOLDGQ5L7GKCCGIH7YXONRW2STRWTB7GMST46JHO5CIJDMJREFHQ, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399227524 (24h25m50.438800644s) [noise=true , address=45.91.116.12:4225 , piece_id=42ALAPORSYG7GMRJP2DQA2QE4RTMFXM6HERWFJ7OTW67R5CMAVLA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399532848 (24h25m50.439105988s) [noise=true , address=45.91.116.4:4939 , piece_id=YLPPJVHYZSYOWGADH34FJPINCGVK2OVROJOJNLKM5QQC36RYLV7Q, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.399842581 (24h25m50.439415701s) [noise=true , address=45.91.116.7:4011 , piece_id=MWJRC3MGEI2HTXB7M4BI2BJ6BLKLUOGU4H7N75PMXDRXEBUQIKEA, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:14.400212709 (24h25m50.439785829s) [noise=true , address=45.91.116.10:4081 , piece_id=WPZWZFZMVWESQHZ5UOGBA4BC3TJAF72HWMIHBFLM2UATWMZ2LZ7A, seg_pos={0 0}, upload=21526]: Uploading piece... Sep 7 13:54:16.270153320 (24h25m52.30972644s) [upload=21526]: Next returned nil segment Sep 7 13:54:16.270172167 (24h25m52.309745277s) [upload=21526]: Waiting for error group managing segments... Sep 7 13:54:16.614864675 (24h25m52.654437795s) [noise=true , address=45.91.116.10:4081 , piece_id=WPZWZFZMVWESQHZ5UOGBA4BC3TJAF72HWMIHBFLM2UATWMZ2LZ7A, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.618689394 (24h25m52.658262514s) [noise=true , address=45.91.116.5:4298 , piece_id=SH6JH5DCRQNI4EVS3XJJGBDJDRZGRCTZBH5BGB5RFE6RK3LKQ2JA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.629888283 (24h25m52.669461403s) [noise=true , address=45.91.116.4:4937 , piece_id=DA5ACPLN3VDN4HPEHDWFLLB5QVWSYSGNW67QG74Z7Y6UJHRNC3MA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.633772636 (24h25m52.673345756s) [noise=true , address=77.90.31.12:4592 , piece_id=CAKOSDXAXI5VWJENCCIWESXPLPSEWYDXA2DWHX3B5WWJB34BTHJA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.637705171 (24h25m52.677278291s) [noise=true , address=45.91.116.7:4011 , piece_id=MWJRC3MGEI2HTXB7M4BI2BJ6BLKLUOGU4H7N75PMXDRXEBUQIKEA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.661186964 (24h25m52.700760084s) [noise=true , address=45.91.116.11:4119 , piece_id=GMBYKDD3FOA2IUETFYO7EAB3GK4OIQC7BFGXMHMO5ZUNH4CFYLKQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.671744326 (24h25m52.711317446s) [noise=true , address=45.91.116.4:4939 , piece_id=YLPPJVHYZSYOWGADH34FJPINCGVK2OVROJOJNLKM5QQC36RYLV7Q, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.682091876 (24h25m52.721664986s) [noise=true , address=45.91.116.12:4225 , piece_id=42ALAPORSYG7GMRJP2DQA2QE4RTMFXM6HERWFJ7OTW67R5CMAVLA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.695192555 (24h25m52.734765675s) [noise=true , address=77.90.31.12:4589 , piece_id=SOLDGQ5L7GKCCGIH7YXONRW2STRWTB7GMST46JHO5CIJDMJREFHQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.704359946 (24h25m52.743933066s) [noise=true , address=45.91.116.10:4083 , piece_id=7RVV6MCA3LSQNL3ZAVNS7G5D3XPZORFXR5MFF37S5DRTVZUVC23Q, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.712230787 (24h25m52.751803907s) [noise=true , address=77.90.31.13:4628 , piece_id=GIRSHGXYPEO5E43IOGSX4A3WEWJD4B7AUOYAH2LSJF5VEHOMEPDA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.719294514 (24h25m52.758867624s) [noise=true , address=77.90.31.15:4698 , piece_id=OMXHZLTUBYFTLBOCFB75NAGPCWWTMDFJIKT6MJ4DP4AXXYI72SIQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.726957527 (24h25m52.766530657s) [noise=true , address=45.91.116.8:4047 , piece_id=7XOVSYGGKYMQW3HKZG4BIO3F53QM424E45ITI5EPYXORPWJ3WHHQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: <nil> Sep 7 13:54:16.726985581 (24h25m52.766558711s) [seg_pos={0 0}, upload=21526]: Segment reached optimal threshold of 13 pieces. Sep 7 13:54:16.727185895 (24h25m52.766759015s) [noise=true , address=77.90.31.14:4662 , piece_id=XCG4JZTQ6FAXX2XJDKCBZ3YUIUSDKU3B6CAMSLU6SRQZXMLNMHAQ, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45126->77.90.31.14:4662: use of closed network connection Sep 7 13:54:16.727325081 (24h25m52.766898191s) [noise=true , address=77.90.31.13:4627 , piece_id=Z4EDCR2HRPPS4RXAFFDEZHCKRKFL4ABQHN4CZMGYKZG6YJW2NKEA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:46560->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 7 13:54:16.727358625 (24h25m52.766931745s) [noise=true , address=45.91.116.6:4975 , piece_id=QYPTGUH4F4MVLWAMXYQLDYZ3VQ7HGZQELZXKX2VENVFFLMVIG4DA, seg_pos={0 0}, upload=21526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44736->45.91.116.6:4975: use of closed network connection Sep 7 13:54:16.727414582 (24h25m52.766987702s) [seg_pos={0 0}, upload=21526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:54:16.727508321 (24h25m52.767081431s) [upload=21526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:54:16.727512310 (24h25m52.76708542s) [upload=21526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:54:17.098909968 (24h25m53.138483088s) [upload=21526]: Done uploading segments... Sep 7 13:54:17.098927803 (24h25m53.138500923s) [upload=21526]: Upload finished. err: <nil> Duration: 30.00134842s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.258067219s FAIL (2025-09-07 13:53:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.258089622s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.X] duration: 30.002966176s FAIL (2025-09-07 13:53:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:53:01.293287117 (24h24m37.332860247s) [upload=21508]: Starting upload Sep 7 13:53:01.293383351 (24h24m37.332956471s) [upload=21508]: Done starting upload Sep 7 13:53:01.293501447 (24h24m37.333074617s) [upload=21508]: Uploading segments... Sep 7 13:53:01.293739874 (24h24m37.333313014s) [upload=21508]: Got next segment. Inline: false Sep 7 13:53:01.293781012 (24h24m37.333354162s) [upload=21508]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:53:01.293787344 (24h24m37.333360464s) [upload=21508]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:53:01.541179806 (24h24m37.580752936s) [seg_pos={0 0}, upload=21508]: Begin upload segment... Sep 7 13:53:01.541235082 (24h24m37.580808202s) [seg_pos={0 0}, upload=21508]: Done begin upload segment. Sep 7 13:53:01.541267093 (24h24m37.580840213s) [noise=true , address=45.91.116.6:4973 , piece_id=FQ4YIAT5JIP267XE7EJXJZMTD47G6G25EWTLJY3RSR4MJVXNC7UQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.541641189 (24h24m37.581214309s) [noise=true , address=45.91.116.7:4009 , piece_id=V4UN3ETVXNTCRLR6OKPFW6CN2PFPPFOD4RP7JXPGBNBN7TBMM5WQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.541921646 (24h24m37.581494766s) [noise=true , address=45.91.116.14:4190 , piece_id=FQLJWQMVBWFKUMXY4BLRXPMUHNMQK4MVSS57F5AXEEZ45E4Y3MPQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.542161264 (24h24m37.581734385s) [noise=true , address=77.90.31.15:4697 , piece_id=S47Z33CW72VE4CASBRYZYLKVZAWRBM5VDRYC4DWI6UBFOVKJ47FQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.542242219 (24h24m37.581815339s) [noise=true , address=77.90.31.15:4698 , piece_id=OO7AALEAAJJCDFJXF36FLIO6YLLFHGMDN7ZJ46X2IPJ3JG2TSNNQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.542552423 (24h24m37.582125543s) [noise=true , address=45.91.116.5:4297 , piece_id=3Z6RTZPQG3IB27OD3SNBA33FTDWKQOPVVWJ3MYAK3FES3TVIG2NA, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.542932360 (24h24m37.58250548s) [noise=true , address=77.90.31.10:4517 , piece_id=PLREKJX4CI6ZUPDPUBGM34LMWY42NIMH6UUMUQ7CF4QPKJPFLCFQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.544162744 (24h24m37.583735864s) [noise=true , address=45.91.116.9:4263 , piece_id=NIZE33YKHEAIINBEUMCMYNEPC7QTVKZKNLQUJAFFGXKP37A2IRXA, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.544453000 (24h24m37.58402613s) [noise=true , address=77.90.31.16:4734 , piece_id=DYJTQ4G7UOHWTOEU5VUTVKPMDBXBVSAT2GKNCXXCK5AKM2T3BEGA, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.544709831 (24h24m37.584282951s) [noise=true , address=45.91.116.11:4118 , piece_id=UH4CMEXVGZI676OR4ZZOMIAVPJBUMCHJVWBYOMJEO3HSDSUMDA5Q, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.544958437 (24h24m37.584531557s) [noise=true , address=77.90.31.14:4663 , piece_id=UNKDQXNWIHJAUY7AL3QSOLI7FUA7XTJSJEZVYEJHYLSQGBQOQICQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.545240606 (24h24m37.584813736s) [noise=true , address=45.91.116.5:4298 , piece_id=A4IFY53CCH4B5Q6O7SQXORZQXLZNYHHTJNZTH3WTSXJ5USIXQ55Q, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.545475656 (24h24m37.585048776s) [noise=true , address=45.91.116.13:4155 , piece_id=E3GZTNG2J7X3GID4IELQUXUUDMAGLBWSLXZO2IOS2UUZACSDUH7Q, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.545690738 (24h24m37.585263858s) [noise=true , address=77.90.31.18:4807 , piece_id=UW3VCRVRQPHDR62GWCAMICRXTICLUAFDXD5GS7ENTMUKDTUDQUFQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.545941017 (24h24m37.585514137s) [noise=true , address=45.91.116.13:4154 , piece_id=4SB7WCCO2YIZ7UGMBUBI5T3TPGVJDUTSQRHWAA4HYITLHBPAP6CQ, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:01.546308981 (24h24m37.585882102s) [noise=true , address=45.91.116.10:4083 , piece_id=7NIHPTYZIVZA4GCDCGDVBI2TTNDPCAYO54ZFNJCIQZ5PLWUZZQ6A, seg_pos={0 0}, upload=21508]: Uploading piece... Sep 7 13:53:03.398517661 (24h24m39.438090781s) [upload=21508]: Next returned nil segment Sep 7 13:53:03.398554983 (24h24m39.438128103s) [upload=21508]: Waiting for error group managing segments... Sep 7 13:53:03.710659423 (24h24m39.750232543s) [noise=true , address=45.91.116.13:4154 , piece_id=4SB7WCCO2YIZ7UGMBUBI5T3TPGVJDUTSQRHWAA4HYITLHBPAP6CQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.783190403 (24h24m39.822763533s) [noise=true , address=77.90.31.15:4697 , piece_id=S47Z33CW72VE4CASBRYZYLKVZAWRBM5VDRYC4DWI6UBFOVKJ47FQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.785393408 (24h24m39.824966528s) [noise=true , address=77.90.31.15:4698 , piece_id=OO7AALEAAJJCDFJXF36FLIO6YLLFHGMDN7ZJ46X2IPJ3JG2TSNNQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.786224990 (24h24m39.82579811s) [noise=true , address=77.90.31.16:4734 , piece_id=DYJTQ4G7UOHWTOEU5VUTVKPMDBXBVSAT2GKNCXXCK5AKM2T3BEGA, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.800227905 (24h24m39.839801025s) [noise=true , address=45.91.116.9:4263 , piece_id=NIZE33YKHEAIINBEUMCMYNEPC7QTVKZKNLQUJAFFGXKP37A2IRXA, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.800651375 (24h24m39.840224505s) [noise=true , address=45.91.116.14:4190 , piece_id=FQLJWQMVBWFKUMXY4BLRXPMUHNMQK4MVSS57F5AXEEZ45E4Y3MPQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.811906943 (24h24m39.851480063s) [noise=true , address=77.90.31.14:4663 , piece_id=UNKDQXNWIHJAUY7AL3QSOLI7FUA7XTJSJEZVYEJHYLSQGBQOQICQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.814416184 (24h24m39.853989304s) [noise=true , address=45.91.116.5:4298 , piece_id=A4IFY53CCH4B5Q6O7SQXORZQXLZNYHHTJNZTH3WTSXJ5USIXQ55Q, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.819823130 (24h24m39.85939625s) [noise=true , address=45.91.116.7:4009 , piece_id=V4UN3ETVXNTCRLR6OKPFW6CN2PFPPFOD4RP7JXPGBNBN7TBMM5WQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.831722481 (24h24m39.871295641s) [noise=true , address=77.90.31.18:4807 , piece_id=UW3VCRVRQPHDR62GWCAMICRXTICLUAFDXD5GS7ENTMUKDTUDQUFQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.843975657 (24h24m39.883548807s) [noise=true , address=45.91.116.11:4118 , piece_id=UH4CMEXVGZI676OR4ZZOMIAVPJBUMCHJVWBYOMJEO3HSDSUMDA5Q, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.905217194 (24h24m39.944790314s) [noise=true , address=45.91.116.10:4083 , piece_id=7NIHPTYZIVZA4GCDCGDVBI2TTNDPCAYO54ZFNJCIQZ5PLWUZZQ6A, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.916507860 (24h24m39.95608098s) [noise=true , address=45.91.116.13:4155 , piece_id=E3GZTNG2J7X3GID4IELQUXUUDMAGLBWSLXZO2IOS2UUZACSDUH7Q, seg_pos={0 0}, upload=21508]: Done uploading piece. err: <nil> Sep 7 13:53:03.916525062 (24h24m39.956098182s) [seg_pos={0 0}, upload=21508]: Segment reached optimal threshold of 13 pieces. Sep 7 13:53:03.916613922 (24h24m39.956187042s) [noise=true , address=45.91.116.5:4297 , piece_id=3Z6RTZPQG3IB27OD3SNBA33FTDWKQOPVVWJ3MYAK3FES3TVIG2NA, seg_pos={0 0}, upload=21508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Sep 7 13:53:03.916680540 (24h24m39.95625366s) [noise=true , address=77.90.31.10:4517 , piece_id=PLREKJX4CI6ZUPDPUBGM34LMWY42NIMH6UUMUQ7CF4QPKJPFLCFQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 7 13:53:03.916829956 (24h24m39.956403076s) [noise=true , address=45.91.116.6:4973 , piece_id=FQ4YIAT5JIP267XE7EJXJZMTD47G6G25EWTLJY3RSR4MJVXNC7UQ, seg_pos={0 0}, upload=21508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:44424->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:44424->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44424->45.91.116.6:4973: use of closed network connection Sep 7 13:53:03.916840506 (24h24m39.956413626s) [seg_pos={0 0}, upload=21508]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:53:03.916921742 (24h24m39.956494862s) [upload=21508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:53:03.916925890 (24h24m39.95649901s) [upload=21508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:53:04.368608687 (24h24m40.408181807s) [upload=21508]: Done uploading segments... Sep 7 13:53:04.368628855 (24h24m40.408201975s) [upload=21508]: Upload finished. err: <nil> Duration: 30.002993749s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.523350334s FAIL (2025-09-07 13:52:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.523376896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X.] duration: 14.28471054s FAIL (2025-09-07 13:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:51:16.913587792 (24h22m52.953160912s) [upload=21485]: Starting upload Sep 7 13:51:16.913649490 (24h22m52.95322261s) [upload=21485]: Done starting upload Sep 7 13:51:16.913714193 (24h22m52.953287313s) [upload=21485]: Uploading segments... Sep 7 13:51:16.913974021 (24h22m52.953547161s) [upload=21485]: Got next segment. Inline: false Sep 7 13:51:16.914094451 (24h22m52.953667601s) [upload=21485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:51:16.914100904 (24h22m52.953674024s) [upload=21485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:51:17.163869582 (24h22m53.203442752s) [seg_pos={0 0}, upload=21485]: Begin upload segment... Sep 7 13:51:17.163922574 (24h22m53.203495694s) [seg_pos={0 0}, upload=21485]: Done begin upload segment. Sep 7 13:51:17.163941410 (24h22m53.20351452s) [noise=true , address=77.90.31.13:4627 , piece_id=MDSNPSNFIAW3WTEU6WYHP24FR2A7I2OCSEON4474HQLMWUDVX6HQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.164312269 (24h22m53.203885389s) [noise=true , address=45.91.116.4:4937 , piece_id=2KU5OAFXMMCGUB3ZXDRK25OIBAXUQKAITI4B2TEEWLORI6BZSIAQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.164598908 (24h22m53.204172028s) [noise=true , address=77.90.31.21:4879 , piece_id=7U23MVX3YPN4MIF4ZFA7S3OOWBUXVDOODTPAULPZ4SYAYV4JN5MA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.164883052 (24h22m53.204456172s) [noise=true , address=45.91.116.6:4973 , piece_id=ZNSFQKMTQXFOSGUAARYHQZ2CU7YPH477GWHDMQERKSUI6E7VVVVA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.165132950 (24h22m53.20470606s) [noise=true , address=45.91.116.11:4119 , piece_id=NR4BNOM6PA56Q7VUP4EYFZPNUHE5NRTI6UTOKO3RWW5YG56B6JVQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.165406643 (24h22m53.204979763s) [noise=true , address=77.90.31.14:4663 , piece_id=QCASIW5SFSQBCG4CEYQNVKSSFGWWRNWQTO6QH7W2345PKEAHYNDA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.165726866 (24h22m53.205299986s) [noise=true , address=77.90.31.14:4662 , piece_id=T4LYZPSE2GLQZHAZNMNGUZNXFT7WMD2KM5T5W3MPHN3XMJRQHVEQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.166985395 (24h22m53.206558515s) [noise=true , address=45.91.116.6:4974 , piece_id=GLY3AMPACCNS3V6FDKYR6VUSAFDVON3A43IKHOAXL6QURJOASW5Q, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.167410518 (24h22m53.206983649s) [noise=true , address=45.91.116.11:4117 , piece_id=Z7AYFH7DA2IGREJKBKRMIW4X6IXEUBOWNCSPTZ6KR6MQHYXCUHKQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.167757081 (24h22m53.207330201s) [noise=true , address=77.90.31.16:4734 , piece_id=DPD35NDYOFULQ6EGF4Y4RNJQHA3SRGDOVLCQOA4SWJZ4CN2DOQMA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168046185 (24h22m53.207619305s) [noise=true , address=77.90.31.21:4877 , piece_id=PD2D5NQFI5KWYJ26RION5W4KWMJEM5ZQ2LX54ENLXQKOQLXADRKA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168340748 (24h22m53.207913868s) [noise=true , address=77.90.31.13:4628 , piece_id=LNLPF2HBEADQMTFMY52UX7TY4QOBJFWSMKPEFH5LOO67JVT7PSQQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168653607 (24h22m53.208226727s) [noise=true , address=45.91.116.10:4081 , piece_id=GIRUANPLP2TGEIR3ALMJYKEYPRQSMP5WAKPKX6TY5JCZY6DR6QKQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168942570 (24h22m53.2085157s) [noise=true , address=77.90.31.12:4589 , piece_id=EJ6DKYVU2JPZYRQ2EMMY4JHJ3MHFCRP4YFORT44Y6IJ7AHYNJGTQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.169206945 (24h22m53.208780075s) [noise=true , address=45.91.116.4:4940 , piece_id=4Q6L2SO7RMGKYOLVQMKZGNSGIYU6O4EB3TTW5ST2NILRUUVO2AEA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.169509424 (24h22m53.209082554s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDNQQV4HBIP6IIEQG2462AVEPO27UWD56S3LC6QMU3747M7HJ4MQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:18.994066777 (24h22m55.033639897s) [upload=21485]: Next returned nil segment Sep 7 13:51:18.994083829 (24h22m55.033656949s) [upload=21485]: Waiting for error group managing segments... Sep 7 13:51:19.343067399 (24h22m55.382640519s) [noise=true , address=45.91.116.6:4973 , piece_id=ZNSFQKMTQXFOSGUAARYHQZ2CU7YPH477GWHDMQERKSUI6E7VVVVA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.352575523 (24h22m55.392148643s) [noise=true , address=77.90.31.12:4589 , piece_id=EJ6DKYVU2JPZYRQ2EMMY4JHJ3MHFCRP4YFORT44Y6IJ7AHYNJGTQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.356735373 (24h22m55.396308493s) [noise=true , address=77.90.31.21:4877 , piece_id=PD2D5NQFI5KWYJ26RION5W4KWMJEM5ZQ2LX54ENLXQKOQLXADRKA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.391734144 (24h22m55.431307254s) [noise=true , address=45.91.116.6:4974 , piece_id=GLY3AMPACCNS3V6FDKYR6VUSAFDVON3A43IKHOAXL6QURJOASW5Q, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.395840412 (24h22m55.435413542s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDNQQV4HBIP6IIEQG2462AVEPO27UWD56S3LC6QMU3747M7HJ4MQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.403164457 (24h22m55.442737577s) [noise=true , address=45.91.116.10:4081 , piece_id=GIRUANPLP2TGEIR3ALMJYKEYPRQSMP5WAKPKX6TY5JCZY6DR6QKQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.412957596 (24h22m55.452530716s) [noise=true , address=45.91.116.11:4119 , piece_id=NR4BNOM6PA56Q7VUP4EYFZPNUHE5NRTI6UTOKO3RWW5YG56B6JVQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.433698554 (24h22m55.473271675s) [noise=true , address=77.90.31.21:4879 , piece_id=7U23MVX3YPN4MIF4ZFA7S3OOWBUXVDOODTPAULPZ4SYAYV4JN5MA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.465229169 (24h22m55.504802289s) [noise=true , address=45.91.116.4:4940 , piece_id=4Q6L2SO7RMGKYOLVQMKZGNSGIYU6O4EB3TTW5ST2NILRUUVO2AEA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.471349270 (24h22m55.51092239s) [noise=true , address=77.90.31.13:4628 , piece_id=LNLPF2HBEADQMTFMY52UX7TY4QOBJFWSMKPEFH5LOO67JVT7PSQQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.476564289 (24h22m55.51613741s) [noise=true , address=77.90.31.14:4662 , piece_id=T4LYZPSE2GLQZHAZNMNGUZNXFT7WMD2KM5T5W3MPHN3XMJRQHVEQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.510542247 (24h22m55.550115367s) [noise=true , address=77.90.31.16:4734 , piece_id=DPD35NDYOFULQ6EGF4Y4RNJQHA3SRGDOVLCQOA4SWJZ4CN2DOQMA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.524159894 (24h22m55.563733024s) [noise=true , address=45.91.116.11:4117 , piece_id=Z7AYFH7DA2IGREJKBKRMIW4X6IXEUBOWNCSPTZ6KR6MQHYXCUHKQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.524184071 (24h22m55.563757191s) [seg_pos={0 0}, upload=21485]: Segment reached optimal threshold of 13 pieces. Sep 7 13:51:19.524320111 (24h22m55.563893221s) [noise=true , address=77.90.31.13:4627 , piece_id=MDSNPSNFIAW3WTEU6WYHP24FR2A7I2OCSEON4474HQLMWUDVX6HQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 7 13:51:19.524368444 (24h22m55.563941564s) [noise=true , address=77.90.31.14:4663 , piece_id=QCASIW5SFSQBCG4CEYQNVKSSFGWWRNWQTO6QH7W2345PKEAHYNDA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:53270->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:53270->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53270->77.90.31.14:4663: use of closed network connection Sep 7 13:51:19.524397870 (24h22m55.56397099s) [noise=true , address=45.91.116.4:4937 , piece_id=2KU5OAFXMMCGUB3ZXDRK25OIBAXUQKAITI4B2TEEWLORI6BZSIAQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:47986->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:47986->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47986->45.91.116.4:4937: use of closed network connection Sep 7 13:51:19.524406497 (24h22m55.563979607s) [seg_pos={0 0}, upload=21485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:51:19.524478113 (24h22m55.564051233s) [upload=21485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:51:19.524483774 (24h22m55.564056894s) [upload=21485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:51:19.825865407 (24h22m55.865438527s) [upload=21485]: Done uploading segments... Sep 7 13:51:19.825892358 (24h22m55.865465488s) [upload=21485]: Upload finished. err: <nil> Duration: 14.284756689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..X] duration: 30.001364442s FAIL (2025-09-07 13:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:51:16.913587792 (24h22m52.953160912s) [upload=21485]: Starting upload Sep 7 13:51:16.913649490 (24h22m52.95322261s) [upload=21485]: Done starting upload Sep 7 13:51:16.913714193 (24h22m52.953287313s) [upload=21485]: Uploading segments... Sep 7 13:51:16.913974021 (24h22m52.953547161s) [upload=21485]: Got next segment. Inline: false Sep 7 13:51:16.914094451 (24h22m52.953667601s) [upload=21485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:51:16.914100904 (24h22m52.953674024s) [upload=21485]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:51:17.163869582 (24h22m53.203442752s) [seg_pos={0 0}, upload=21485]: Begin upload segment... Sep 7 13:51:17.163922574 (24h22m53.203495694s) [seg_pos={0 0}, upload=21485]: Done begin upload segment. Sep 7 13:51:17.163941410 (24h22m53.20351452s) [noise=true , address=77.90.31.13:4627 , piece_id=MDSNPSNFIAW3WTEU6WYHP24FR2A7I2OCSEON4474HQLMWUDVX6HQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.164312269 (24h22m53.203885389s) [noise=true , address=45.91.116.4:4937 , piece_id=2KU5OAFXMMCGUB3ZXDRK25OIBAXUQKAITI4B2TEEWLORI6BZSIAQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.164598908 (24h22m53.204172028s) [noise=true , address=77.90.31.21:4879 , piece_id=7U23MVX3YPN4MIF4ZFA7S3OOWBUXVDOODTPAULPZ4SYAYV4JN5MA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.164883052 (24h22m53.204456172s) [noise=true , address=45.91.116.6:4973 , piece_id=ZNSFQKMTQXFOSGUAARYHQZ2CU7YPH477GWHDMQERKSUI6E7VVVVA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.165132950 (24h22m53.20470606s) [noise=true , address=45.91.116.11:4119 , piece_id=NR4BNOM6PA56Q7VUP4EYFZPNUHE5NRTI6UTOKO3RWW5YG56B6JVQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.165406643 (24h22m53.204979763s) [noise=true , address=77.90.31.14:4663 , piece_id=QCASIW5SFSQBCG4CEYQNVKSSFGWWRNWQTO6QH7W2345PKEAHYNDA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.165726866 (24h22m53.205299986s) [noise=true , address=77.90.31.14:4662 , piece_id=T4LYZPSE2GLQZHAZNMNGUZNXFT7WMD2KM5T5W3MPHN3XMJRQHVEQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.166985395 (24h22m53.206558515s) [noise=true , address=45.91.116.6:4974 , piece_id=GLY3AMPACCNS3V6FDKYR6VUSAFDVON3A43IKHOAXL6QURJOASW5Q, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.167410518 (24h22m53.206983649s) [noise=true , address=45.91.116.11:4117 , piece_id=Z7AYFH7DA2IGREJKBKRMIW4X6IXEUBOWNCSPTZ6KR6MQHYXCUHKQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.167757081 (24h22m53.207330201s) [noise=true , address=77.90.31.16:4734 , piece_id=DPD35NDYOFULQ6EGF4Y4RNJQHA3SRGDOVLCQOA4SWJZ4CN2DOQMA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168046185 (24h22m53.207619305s) [noise=true , address=77.90.31.21:4877 , piece_id=PD2D5NQFI5KWYJ26RION5W4KWMJEM5ZQ2LX54ENLXQKOQLXADRKA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168340748 (24h22m53.207913868s) [noise=true , address=77.90.31.13:4628 , piece_id=LNLPF2HBEADQMTFMY52UX7TY4QOBJFWSMKPEFH5LOO67JVT7PSQQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168653607 (24h22m53.208226727s) [noise=true , address=45.91.116.10:4081 , piece_id=GIRUANPLP2TGEIR3ALMJYKEYPRQSMP5WAKPKX6TY5JCZY6DR6QKQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.168942570 (24h22m53.2085157s) [noise=true , address=77.90.31.12:4589 , piece_id=EJ6DKYVU2JPZYRQ2EMMY4JHJ3MHFCRP4YFORT44Y6IJ7AHYNJGTQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.169206945 (24h22m53.208780075s) [noise=true , address=45.91.116.4:4940 , piece_id=4Q6L2SO7RMGKYOLVQMKZGNSGIYU6O4EB3TTW5ST2NILRUUVO2AEA, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:17.169509424 (24h22m53.209082554s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDNQQV4HBIP6IIEQG2462AVEPO27UWD56S3LC6QMU3747M7HJ4MQ, seg_pos={0 0}, upload=21485]: Uploading piece... Sep 7 13:51:18.994066777 (24h22m55.033639897s) [upload=21485]: Next returned nil segment Sep 7 13:51:18.994083829 (24h22m55.033656949s) [upload=21485]: Waiting for error group managing segments... Sep 7 13:51:19.343067399 (24h22m55.382640519s) [noise=true , address=45.91.116.6:4973 , piece_id=ZNSFQKMTQXFOSGUAARYHQZ2CU7YPH477GWHDMQERKSUI6E7VVVVA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.352575523 (24h22m55.392148643s) [noise=true , address=77.90.31.12:4589 , piece_id=EJ6DKYVU2JPZYRQ2EMMY4JHJ3MHFCRP4YFORT44Y6IJ7AHYNJGTQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.356735373 (24h22m55.396308493s) [noise=true , address=77.90.31.21:4877 , piece_id=PD2D5NQFI5KWYJ26RION5W4KWMJEM5ZQ2LX54ENLXQKOQLXADRKA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.391734144 (24h22m55.431307254s) [noise=true , address=45.91.116.6:4974 , piece_id=GLY3AMPACCNS3V6FDKYR6VUSAFDVON3A43IKHOAXL6QURJOASW5Q, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.395840412 (24h22m55.435413542s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDNQQV4HBIP6IIEQG2462AVEPO27UWD56S3LC6QMU3747M7HJ4MQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.403164457 (24h22m55.442737577s) [noise=true , address=45.91.116.10:4081 , piece_id=GIRUANPLP2TGEIR3ALMJYKEYPRQSMP5WAKPKX6TY5JCZY6DR6QKQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.412957596 (24h22m55.452530716s) [noise=true , address=45.91.116.11:4119 , piece_id=NR4BNOM6PA56Q7VUP4EYFZPNUHE5NRTI6UTOKO3RWW5YG56B6JVQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.433698554 (24h22m55.473271675s) [noise=true , address=77.90.31.21:4879 , piece_id=7U23MVX3YPN4MIF4ZFA7S3OOWBUXVDOODTPAULPZ4SYAYV4JN5MA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.465229169 (24h22m55.504802289s) [noise=true , address=45.91.116.4:4940 , piece_id=4Q6L2SO7RMGKYOLVQMKZGNSGIYU6O4EB3TTW5ST2NILRUUVO2AEA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.471349270 (24h22m55.51092239s) [noise=true , address=77.90.31.13:4628 , piece_id=LNLPF2HBEADQMTFMY52UX7TY4QOBJFWSMKPEFH5LOO67JVT7PSQQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.476564289 (24h22m55.51613741s) [noise=true , address=77.90.31.14:4662 , piece_id=T4LYZPSE2GLQZHAZNMNGUZNXFT7WMD2KM5T5W3MPHN3XMJRQHVEQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.510542247 (24h22m55.550115367s) [noise=true , address=77.90.31.16:4734 , piece_id=DPD35NDYOFULQ6EGF4Y4RNJQHA3SRGDOVLCQOA4SWJZ4CN2DOQMA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.524159894 (24h22m55.563733024s) [noise=true , address=45.91.116.11:4117 , piece_id=Z7AYFH7DA2IGREJKBKRMIW4X6IXEUBOWNCSPTZ6KR6MQHYXCUHKQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: <nil> Sep 7 13:51:19.524184071 (24h22m55.563757191s) [seg_pos={0 0}, upload=21485]: Segment reached optimal threshold of 13 pieces. Sep 7 13:51:19.524320111 (24h22m55.563893221s) [noise=true , address=77.90.31.13:4627 , piece_id=MDSNPSNFIAW3WTEU6WYHP24FR2A7I2OCSEON4474HQLMWUDVX6HQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 7 13:51:19.524368444 (24h22m55.563941564s) [noise=true , address=77.90.31.14:4663 , piece_id=QCASIW5SFSQBCG4CEYQNVKSSFGWWRNWQTO6QH7W2345PKEAHYNDA, seg_pos={0 0}, upload=21485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:53270->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:53270->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53270->77.90.31.14:4663: use of closed network connection Sep 7 13:51:19.524397870 (24h22m55.56397099s) [noise=true , address=45.91.116.4:4937 , piece_id=2KU5OAFXMMCGUB3ZXDRK25OIBAXUQKAITI4B2TEEWLORI6BZSIAQ, seg_pos={0 0}, upload=21485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:47986->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:47986->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47986->45.91.116.4:4937: use of closed network connection Sep 7 13:51:19.524406497 (24h22m55.563979607s) [seg_pos={0 0}, upload=21485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:51:19.524478113 (24h22m55.564051233s) [upload=21485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:51:19.524483774 (24h22m55.564056894s) [upload=21485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:51:19.825865407 (24h22m55.865438527s) [upload=21485]: Done uploading segments... Sep 7 13:51:19.825892358 (24h22m55.865465488s) [upload=21485]: Upload finished. err: <nil> Duration: 30.001387406s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.810521545s FAIL (2025-09-07 13:51:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.810562544s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X..] duration: 16.1113589s FAIL (2025-09-07 13:50:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:49:29.945868138 (24h21m5.985441258s) [upload=21459]: Starting upload Sep 7 13:49:29.945928023 (24h21m5.985501143s) [upload=21459]: Done starting upload Sep 7 13:49:29.946063833 (24h21m5.985636953s) [upload=21459]: Uploading segments... Sep 7 13:49:29.946218588 (24h21m5.985791708s) [upload=21459]: Got next segment. Inline: false Sep 7 13:49:29.946257764 (24h21m5.985830884s) [upload=21459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:49:29.946262813 (24h21m5.985835933s) [upload=21459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:49:30.282847407 (24h21m6.322420527s) [seg_pos={0 0}, upload=21459]: Begin upload segment... Sep 7 13:49:30.282905067 (24h21m6.322478187s) [seg_pos={0 0}, upload=21459]: Done begin upload segment. Sep 7 13:49:30.282921519 (24h21m6.322494639s) [noise=true , address=45.91.116.5:4297 , piece_id=GJU4AYAEF2VSYSWXO4X3R6UMCQUBU6OA2FWGJSKTKNXVYQ2MOXWQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.282938381 (24h21m6.322511501s) [noise=true , address=45.91.116.14:4191 , piece_id=XZN24LW64LQQGKSS6WWFLCQWJFJPVPQB4MUOFN5BKTTIF2S4QVYQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283282700 (24h21m6.32285582s) [noise=true , address=45.91.116.5:4299 , piece_id=OCPZBYGXJV3AAXZDGX3X56EQKLR7ZIHMNZGSVKHLKXSBFMXBTRYQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283368194 (24h21m6.322941314s) [noise=true , address=77.90.31.15:4699 , piece_id=H63TGDNZ3VFRS5FD6AY5X3L7RU2P2HJTNWM2OZMA2YF2GAEZSOIA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283541306 (24h21m6.323114426s) [noise=true , address=45.91.116.12:4226 , piece_id=ZQB5JBOWAAPDJPB22ZPDP3UAFTVFRUEXCBZRP7A722NIRVNPAPTQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283788959 (24h21m6.323362079s) [noise=true , address=77.90.31.16:4734 , piece_id=ALOAOSIUESGZYJOEAJ6OYVGAPWXZYJA7ZGTDIWLPHSBBADL3DCUA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284113029 (24h21m6.323686149s) [noise=true , address=45.91.116.12:4225 , piece_id=PQH5MNGKUOET3JPJERGPU2FR4DRMGJVQF2UUEY35QL36XLGH5MOA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284389689 (24h21m6.323962809s) [noise=true , address=77.90.31.12:4589 , piece_id=FUF7SR2NE6KXG3HA7JBDR2NZWJZW5DEAFTA5ZDXHQ272SXWBWNXQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284647221 (24h21m6.324220341s) [noise=true , address=77.90.31.10:4520 , piece_id=AINVPKJJZJTDZ3NQPS2JYXHH3R7EBKFEHULS2CKUNBUBPUV3X6FQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284914593 (24h21m6.324487713s) [noise=true , address=77.90.31.16:4733 , piece_id=IPBOLHWLYZEG6QFBYQUAYCI3QJBU3GGWM5NRKE2NH6DPPUALYGHQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.285261748 (24h21m6.324834868s) [noise=true , address=77.90.31.15:4697 , piece_id=MBGC6WJPYTKNXRPY6FHCRQF6KI5ZS5HIFWSLRP2YTSYHS44KONAQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.285595416 (24h21m6.325168536s) [noise=true , address=45.91.116.8:4048 , piece_id=GC7TYYXIZGBSGOS6YSRFC5UUFSVPW7OOVBWKTAJRFGTI6L5YYLMQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286073651 (24h21m6.325646781s) [noise=true , address=77.90.31.13:4628 , piece_id=WIMH65DLK5EEDIQBZBOBQBZP3ANLBSTA3KFFUMKPM3PAIWSIHHDA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286406026 (24h21m6.325979147s) [noise=true , address=77.90.31.19:4843 , piece_id=IBD4LTX2CGXUR4LFD42QYPEZVE3VQGXAKSKQTFX4XSO7HF43S64A, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286687205 (24h21m6.326260325s) [noise=true , address=45.91.116.4:4939 , piece_id=TZFIPLBUVTXQ6NXUBSJYAPAD5M2DU62RKMZ3T2UWVQBBXUYNFFQQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286893790 (24h21m6.32646691s) [noise=true , address=45.91.116.13:4153 , piece_id=2JCVNU2LPJBYFLY5WSVPXA5MKH43SUOUESPIFHVFWF5QE6ST3NUQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:32.176225038 (24h21m8.215798158s) [upload=21459]: Next returned nil segment Sep 7 13:49:32.176258703 (24h21m8.215831823s) [upload=21459]: Waiting for error group managing segments... Sep 7 13:49:32.469436086 (24h21m8.509009206s) [noise=true , address=45.91.116.13:4153 , piece_id=2JCVNU2LPJBYFLY5WSVPXA5MKH43SUOUESPIFHVFWF5QE6ST3NUQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.488165356 (24h21m8.527738476s) [noise=true , address=45.91.116.8:4048 , piece_id=GC7TYYXIZGBSGOS6YSRFC5UUFSVPW7OOVBWKTAJRFGTI6L5YYLMQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.544578975 (24h21m8.584152095s) [noise=true , address=45.91.116.14:4191 , piece_id=XZN24LW64LQQGKSS6WWFLCQWJFJPVPQB4MUOFN5BKTTIF2S4QVYQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.554371102 (24h21m8.593944222s) [noise=true , address=77.90.31.10:4520 , piece_id=AINVPKJJZJTDZ3NQPS2JYXHH3R7EBKFEHULS2CKUNBUBPUV3X6FQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.561871885 (24h21m8.601445005s) [noise=true , address=45.91.116.4:4939 , piece_id=TZFIPLBUVTXQ6NXUBSJYAPAD5M2DU62RKMZ3T2UWVQBBXUYNFFQQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.572645080 (24h21m8.6122182s) [noise=true , address=77.90.31.13:4628 , piece_id=WIMH65DLK5EEDIQBZBOBQBZP3ANLBSTA3KFFUMKPM3PAIWSIHHDA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.573575430 (24h21m8.61314855s) [noise=true , address=77.90.31.16:4733 , piece_id=IPBOLHWLYZEG6QFBYQUAYCI3QJBU3GGWM5NRKE2NH6DPPUALYGHQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.604017051 (24h21m8.643590171s) [noise=true , address=77.90.31.12:4589 , piece_id=FUF7SR2NE6KXG3HA7JBDR2NZWJZW5DEAFTA5ZDXHQ272SXWBWNXQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.611068965 (24h21m8.650642085s) [noise=true , address=77.90.31.15:4699 , piece_id=H63TGDNZ3VFRS5FD6AY5X3L7RU2P2HJTNWM2OZMA2YF2GAEZSOIA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.619810232 (24h21m8.659383352s) [noise=true , address=77.90.31.19:4843 , piece_id=IBD4LTX2CGXUR4LFD42QYPEZVE3VQGXAKSKQTFX4XSO7HF43S64A, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.629721867 (24h21m8.669294987s) [noise=true , address=77.90.31.16:4734 , piece_id=ALOAOSIUESGZYJOEAJ6OYVGAPWXZYJA7ZGTDIWLPHSBBADL3DCUA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632035244 (24h21m8.671608364s) [noise=true , address=77.90.31.15:4697 , piece_id=MBGC6WJPYTKNXRPY6FHCRQF6KI5ZS5HIFWSLRP2YTSYHS44KONAQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632203987 (24h21m8.671777107s) [noise=true , address=45.91.116.5:4297 , piece_id=GJU4AYAEF2VSYSWXO4X3R6UMCQUBU6OA2FWGJSKTKNXVYQ2MOXWQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632273691 (24h21m8.671846821s) [seg_pos={0 0}, upload=21459]: Segment reached optimal threshold of 13 pieces. Sep 7 13:49:32.632485426 (24h21m8.672058556s) [noise=true , address=45.91.116.12:4226 , piece_id=ZQB5JBOWAAPDJPB22ZPDP3UAFTVFRUEXCBZRP7A722NIRVNPAPTQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 7 13:49:32.632538438 (24h21m8.672111568s) [noise=true , address=45.91.116.5:4299 , piece_id=OCPZBYGXJV3AAXZDGX3X56EQKLR7ZIHMNZGSVKHLKXSBFMXBTRYQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 7 13:49:32.632551993 (24h21m8.672125113s) [noise=true , address=45.91.116.12:4225 , piece_id=PQH5MNGKUOET3JPJERGPU2FR4DRMGJVQF2UUEY35QL36XLGH5MOA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection Sep 7 13:49:32.632586359 (24h21m8.672159479s) [seg_pos={0 0}, upload=21459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:49:32.632709044 (24h21m8.672282164s) [upload=21459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:49:32.632713823 (24h21m8.672286943s) [upload=21459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:49:32.964835227 (24h21m9.004408347s) [upload=21459]: Done uploading segments... Sep 7 13:49:32.964858161 (24h21m9.004431281s) [upload=21459]: Upload finished. err: <nil> Duration: 16.111414216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X.] duration: 26.868067685s FAIL (2025-09-07 13:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:49:29.945868138 (24h21m5.985441258s) [upload=21459]: Starting upload Sep 7 13:49:29.945928023 (24h21m5.985501143s) [upload=21459]: Done starting upload Sep 7 13:49:29.946063833 (24h21m5.985636953s) [upload=21459]: Uploading segments... Sep 7 13:49:29.946218588 (24h21m5.985791708s) [upload=21459]: Got next segment. Inline: false Sep 7 13:49:29.946257764 (24h21m5.985830884s) [upload=21459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:49:29.946262813 (24h21m5.985835933s) [upload=21459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:49:30.282847407 (24h21m6.322420527s) [seg_pos={0 0}, upload=21459]: Begin upload segment... Sep 7 13:49:30.282905067 (24h21m6.322478187s) [seg_pos={0 0}, upload=21459]: Done begin upload segment. Sep 7 13:49:30.282921519 (24h21m6.322494639s) [noise=true , address=45.91.116.5:4297 , piece_id=GJU4AYAEF2VSYSWXO4X3R6UMCQUBU6OA2FWGJSKTKNXVYQ2MOXWQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.282938381 (24h21m6.322511501s) [noise=true , address=45.91.116.14:4191 , piece_id=XZN24LW64LQQGKSS6WWFLCQWJFJPVPQB4MUOFN5BKTTIF2S4QVYQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283282700 (24h21m6.32285582s) [noise=true , address=45.91.116.5:4299 , piece_id=OCPZBYGXJV3AAXZDGX3X56EQKLR7ZIHMNZGSVKHLKXSBFMXBTRYQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283368194 (24h21m6.322941314s) [noise=true , address=77.90.31.15:4699 , piece_id=H63TGDNZ3VFRS5FD6AY5X3L7RU2P2HJTNWM2OZMA2YF2GAEZSOIA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283541306 (24h21m6.323114426s) [noise=true , address=45.91.116.12:4226 , piece_id=ZQB5JBOWAAPDJPB22ZPDP3UAFTVFRUEXCBZRP7A722NIRVNPAPTQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283788959 (24h21m6.323362079s) [noise=true , address=77.90.31.16:4734 , piece_id=ALOAOSIUESGZYJOEAJ6OYVGAPWXZYJA7ZGTDIWLPHSBBADL3DCUA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284113029 (24h21m6.323686149s) [noise=true , address=45.91.116.12:4225 , piece_id=PQH5MNGKUOET3JPJERGPU2FR4DRMGJVQF2UUEY35QL36XLGH5MOA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284389689 (24h21m6.323962809s) [noise=true , address=77.90.31.12:4589 , piece_id=FUF7SR2NE6KXG3HA7JBDR2NZWJZW5DEAFTA5ZDXHQ272SXWBWNXQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284647221 (24h21m6.324220341s) [noise=true , address=77.90.31.10:4520 , piece_id=AINVPKJJZJTDZ3NQPS2JYXHH3R7EBKFEHULS2CKUNBUBPUV3X6FQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284914593 (24h21m6.324487713s) [noise=true , address=77.90.31.16:4733 , piece_id=IPBOLHWLYZEG6QFBYQUAYCI3QJBU3GGWM5NRKE2NH6DPPUALYGHQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.285261748 (24h21m6.324834868s) [noise=true , address=77.90.31.15:4697 , piece_id=MBGC6WJPYTKNXRPY6FHCRQF6KI5ZS5HIFWSLRP2YTSYHS44KONAQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.285595416 (24h21m6.325168536s) [noise=true , address=45.91.116.8:4048 , piece_id=GC7TYYXIZGBSGOS6YSRFC5UUFSVPW7OOVBWKTAJRFGTI6L5YYLMQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286073651 (24h21m6.325646781s) [noise=true , address=77.90.31.13:4628 , piece_id=WIMH65DLK5EEDIQBZBOBQBZP3ANLBSTA3KFFUMKPM3PAIWSIHHDA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286406026 (24h21m6.325979147s) [noise=true , address=77.90.31.19:4843 , piece_id=IBD4LTX2CGXUR4LFD42QYPEZVE3VQGXAKSKQTFX4XSO7HF43S64A, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286687205 (24h21m6.326260325s) [noise=true , address=45.91.116.4:4939 , piece_id=TZFIPLBUVTXQ6NXUBSJYAPAD5M2DU62RKMZ3T2UWVQBBXUYNFFQQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286893790 (24h21m6.32646691s) [noise=true , address=45.91.116.13:4153 , piece_id=2JCVNU2LPJBYFLY5WSVPXA5MKH43SUOUESPIFHVFWF5QE6ST3NUQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:32.176225038 (24h21m8.215798158s) [upload=21459]: Next returned nil segment Sep 7 13:49:32.176258703 (24h21m8.215831823s) [upload=21459]: Waiting for error group managing segments... Sep 7 13:49:32.469436086 (24h21m8.509009206s) [noise=true , address=45.91.116.13:4153 , piece_id=2JCVNU2LPJBYFLY5WSVPXA5MKH43SUOUESPIFHVFWF5QE6ST3NUQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.488165356 (24h21m8.527738476s) [noise=true , address=45.91.116.8:4048 , piece_id=GC7TYYXIZGBSGOS6YSRFC5UUFSVPW7OOVBWKTAJRFGTI6L5YYLMQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.544578975 (24h21m8.584152095s) [noise=true , address=45.91.116.14:4191 , piece_id=XZN24LW64LQQGKSS6WWFLCQWJFJPVPQB4MUOFN5BKTTIF2S4QVYQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.554371102 (24h21m8.593944222s) [noise=true , address=77.90.31.10:4520 , piece_id=AINVPKJJZJTDZ3NQPS2JYXHH3R7EBKFEHULS2CKUNBUBPUV3X6FQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.561871885 (24h21m8.601445005s) [noise=true , address=45.91.116.4:4939 , piece_id=TZFIPLBUVTXQ6NXUBSJYAPAD5M2DU62RKMZ3T2UWVQBBXUYNFFQQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.572645080 (24h21m8.6122182s) [noise=true , address=77.90.31.13:4628 , piece_id=WIMH65DLK5EEDIQBZBOBQBZP3ANLBSTA3KFFUMKPM3PAIWSIHHDA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.573575430 (24h21m8.61314855s) [noise=true , address=77.90.31.16:4733 , piece_id=IPBOLHWLYZEG6QFBYQUAYCI3QJBU3GGWM5NRKE2NH6DPPUALYGHQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.604017051 (24h21m8.643590171s) [noise=true , address=77.90.31.12:4589 , piece_id=FUF7SR2NE6KXG3HA7JBDR2NZWJZW5DEAFTA5ZDXHQ272SXWBWNXQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.611068965 (24h21m8.650642085s) [noise=true , address=77.90.31.15:4699 , piece_id=H63TGDNZ3VFRS5FD6AY5X3L7RU2P2HJTNWM2OZMA2YF2GAEZSOIA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.619810232 (24h21m8.659383352s) [noise=true , address=77.90.31.19:4843 , piece_id=IBD4LTX2CGXUR4LFD42QYPEZVE3VQGXAKSKQTFX4XSO7HF43S64A, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.629721867 (24h21m8.669294987s) [noise=true , address=77.90.31.16:4734 , piece_id=ALOAOSIUESGZYJOEAJ6OYVGAPWXZYJA7ZGTDIWLPHSBBADL3DCUA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632035244 (24h21m8.671608364s) [noise=true , address=77.90.31.15:4697 , piece_id=MBGC6WJPYTKNXRPY6FHCRQF6KI5ZS5HIFWSLRP2YTSYHS44KONAQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632203987 (24h21m8.671777107s) [noise=true , address=45.91.116.5:4297 , piece_id=GJU4AYAEF2VSYSWXO4X3R6UMCQUBU6OA2FWGJSKTKNXVYQ2MOXWQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632273691 (24h21m8.671846821s) [seg_pos={0 0}, upload=21459]: Segment reached optimal threshold of 13 pieces. Sep 7 13:49:32.632485426 (24h21m8.672058556s) [noise=true , address=45.91.116.12:4226 , piece_id=ZQB5JBOWAAPDJPB22ZPDP3UAFTVFRUEXCBZRP7A722NIRVNPAPTQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 7 13:49:32.632538438 (24h21m8.672111568s) [noise=true , address=45.91.116.5:4299 , piece_id=OCPZBYGXJV3AAXZDGX3X56EQKLR7ZIHMNZGSVKHLKXSBFMXBTRYQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 7 13:49:32.632551993 (24h21m8.672125113s) [noise=true , address=45.91.116.12:4225 , piece_id=PQH5MNGKUOET3JPJERGPU2FR4DRMGJVQF2UUEY35QL36XLGH5MOA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection Sep 7 13:49:32.632586359 (24h21m8.672159479s) [seg_pos={0 0}, upload=21459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:49:32.632709044 (24h21m8.672282164s) [upload=21459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:49:32.632713823 (24h21m8.672286943s) [upload=21459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:49:32.964835227 (24h21m9.004408347s) [upload=21459]: Done uploading segments... Sep 7 13:49:32.964858161 (24h21m9.004431281s) [upload=21459]: Upload finished. err: <nil> Duration: 26.868093626s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.224578175s FAIL (2025-09-07 13:50:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.224598746s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........X] duration: 30.002152629s FAIL (2025-09-07 13:49:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:49:29.945868138 (24h21m5.985441258s) [upload=21459]: Starting upload Sep 7 13:49:29.945928023 (24h21m5.985501143s) [upload=21459]: Done starting upload Sep 7 13:49:29.946063833 (24h21m5.985636953s) [upload=21459]: Uploading segments... Sep 7 13:49:29.946218588 (24h21m5.985791708s) [upload=21459]: Got next segment. Inline: false Sep 7 13:49:29.946257764 (24h21m5.985830884s) [upload=21459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:49:29.946262813 (24h21m5.985835933s) [upload=21459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:49:30.282847407 (24h21m6.322420527s) [seg_pos={0 0}, upload=21459]: Begin upload segment... Sep 7 13:49:30.282905067 (24h21m6.322478187s) [seg_pos={0 0}, upload=21459]: Done begin upload segment. Sep 7 13:49:30.282921519 (24h21m6.322494639s) [noise=true , address=45.91.116.5:4297 , piece_id=GJU4AYAEF2VSYSWXO4X3R6UMCQUBU6OA2FWGJSKTKNXVYQ2MOXWQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.282938381 (24h21m6.322511501s) [noise=true , address=45.91.116.14:4191 , piece_id=XZN24LW64LQQGKSS6WWFLCQWJFJPVPQB4MUOFN5BKTTIF2S4QVYQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283282700 (24h21m6.32285582s) [noise=true , address=45.91.116.5:4299 , piece_id=OCPZBYGXJV3AAXZDGX3X56EQKLR7ZIHMNZGSVKHLKXSBFMXBTRYQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283368194 (24h21m6.322941314s) [noise=true , address=77.90.31.15:4699 , piece_id=H63TGDNZ3VFRS5FD6AY5X3L7RU2P2HJTNWM2OZMA2YF2GAEZSOIA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283541306 (24h21m6.323114426s) [noise=true , address=45.91.116.12:4226 , piece_id=ZQB5JBOWAAPDJPB22ZPDP3UAFTVFRUEXCBZRP7A722NIRVNPAPTQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.283788959 (24h21m6.323362079s) [noise=true , address=77.90.31.16:4734 , piece_id=ALOAOSIUESGZYJOEAJ6OYVGAPWXZYJA7ZGTDIWLPHSBBADL3DCUA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284113029 (24h21m6.323686149s) [noise=true , address=45.91.116.12:4225 , piece_id=PQH5MNGKUOET3JPJERGPU2FR4DRMGJVQF2UUEY35QL36XLGH5MOA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284389689 (24h21m6.323962809s) [noise=true , address=77.90.31.12:4589 , piece_id=FUF7SR2NE6KXG3HA7JBDR2NZWJZW5DEAFTA5ZDXHQ272SXWBWNXQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284647221 (24h21m6.324220341s) [noise=true , address=77.90.31.10:4520 , piece_id=AINVPKJJZJTDZ3NQPS2JYXHH3R7EBKFEHULS2CKUNBUBPUV3X6FQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.284914593 (24h21m6.324487713s) [noise=true , address=77.90.31.16:4733 , piece_id=IPBOLHWLYZEG6QFBYQUAYCI3QJBU3GGWM5NRKE2NH6DPPUALYGHQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.285261748 (24h21m6.324834868s) [noise=true , address=77.90.31.15:4697 , piece_id=MBGC6WJPYTKNXRPY6FHCRQF6KI5ZS5HIFWSLRP2YTSYHS44KONAQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.285595416 (24h21m6.325168536s) [noise=true , address=45.91.116.8:4048 , piece_id=GC7TYYXIZGBSGOS6YSRFC5UUFSVPW7OOVBWKTAJRFGTI6L5YYLMQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286073651 (24h21m6.325646781s) [noise=true , address=77.90.31.13:4628 , piece_id=WIMH65DLK5EEDIQBZBOBQBZP3ANLBSTA3KFFUMKPM3PAIWSIHHDA, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286406026 (24h21m6.325979147s) [noise=true , address=77.90.31.19:4843 , piece_id=IBD4LTX2CGXUR4LFD42QYPEZVE3VQGXAKSKQTFX4XSO7HF43S64A, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286687205 (24h21m6.326260325s) [noise=true , address=45.91.116.4:4939 , piece_id=TZFIPLBUVTXQ6NXUBSJYAPAD5M2DU62RKMZ3T2UWVQBBXUYNFFQQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:30.286893790 (24h21m6.32646691s) [noise=true , address=45.91.116.13:4153 , piece_id=2JCVNU2LPJBYFLY5WSVPXA5MKH43SUOUESPIFHVFWF5QE6ST3NUQ, seg_pos={0 0}, upload=21459]: Uploading piece... Sep 7 13:49:32.176225038 (24h21m8.215798158s) [upload=21459]: Next returned nil segment Sep 7 13:49:32.176258703 (24h21m8.215831823s) [upload=21459]: Waiting for error group managing segments... Sep 7 13:49:32.469436086 (24h21m8.509009206s) [noise=true , address=45.91.116.13:4153 , piece_id=2JCVNU2LPJBYFLY5WSVPXA5MKH43SUOUESPIFHVFWF5QE6ST3NUQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.488165356 (24h21m8.527738476s) [noise=true , address=45.91.116.8:4048 , piece_id=GC7TYYXIZGBSGOS6YSRFC5UUFSVPW7OOVBWKTAJRFGTI6L5YYLMQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.544578975 (24h21m8.584152095s) [noise=true , address=45.91.116.14:4191 , piece_id=XZN24LW64LQQGKSS6WWFLCQWJFJPVPQB4MUOFN5BKTTIF2S4QVYQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.554371102 (24h21m8.593944222s) [noise=true , address=77.90.31.10:4520 , piece_id=AINVPKJJZJTDZ3NQPS2JYXHH3R7EBKFEHULS2CKUNBUBPUV3X6FQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.561871885 (24h21m8.601445005s) [noise=true , address=45.91.116.4:4939 , piece_id=TZFIPLBUVTXQ6NXUBSJYAPAD5M2DU62RKMZ3T2UWVQBBXUYNFFQQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.572645080 (24h21m8.6122182s) [noise=true , address=77.90.31.13:4628 , piece_id=WIMH65DLK5EEDIQBZBOBQBZP3ANLBSTA3KFFUMKPM3PAIWSIHHDA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.573575430 (24h21m8.61314855s) [noise=true , address=77.90.31.16:4733 , piece_id=IPBOLHWLYZEG6QFBYQUAYCI3QJBU3GGWM5NRKE2NH6DPPUALYGHQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.604017051 (24h21m8.643590171s) [noise=true , address=77.90.31.12:4589 , piece_id=FUF7SR2NE6KXG3HA7JBDR2NZWJZW5DEAFTA5ZDXHQ272SXWBWNXQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.611068965 (24h21m8.650642085s) [noise=true , address=77.90.31.15:4699 , piece_id=H63TGDNZ3VFRS5FD6AY5X3L7RU2P2HJTNWM2OZMA2YF2GAEZSOIA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.619810232 (24h21m8.659383352s) [noise=true , address=77.90.31.19:4843 , piece_id=IBD4LTX2CGXUR4LFD42QYPEZVE3VQGXAKSKQTFX4XSO7HF43S64A, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.629721867 (24h21m8.669294987s) [noise=true , address=77.90.31.16:4734 , piece_id=ALOAOSIUESGZYJOEAJ6OYVGAPWXZYJA7ZGTDIWLPHSBBADL3DCUA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632035244 (24h21m8.671608364s) [noise=true , address=77.90.31.15:4697 , piece_id=MBGC6WJPYTKNXRPY6FHCRQF6KI5ZS5HIFWSLRP2YTSYHS44KONAQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632203987 (24h21m8.671777107s) [noise=true , address=45.91.116.5:4297 , piece_id=GJU4AYAEF2VSYSWXO4X3R6UMCQUBU6OA2FWGJSKTKNXVYQ2MOXWQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: <nil> Sep 7 13:49:32.632273691 (24h21m8.671846821s) [seg_pos={0 0}, upload=21459]: Segment reached optimal threshold of 13 pieces. Sep 7 13:49:32.632485426 (24h21m8.672058556s) [noise=true , address=45.91.116.12:4226 , piece_id=ZQB5JBOWAAPDJPB22ZPDP3UAFTVFRUEXCBZRP7A722NIRVNPAPTQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 7 13:49:32.632538438 (24h21m8.672111568s) [noise=true , address=45.91.116.5:4299 , piece_id=OCPZBYGXJV3AAXZDGX3X56EQKLR7ZIHMNZGSVKHLKXSBFMXBTRYQ, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 7 13:49:32.632551993 (24h21m8.672125113s) [noise=true , address=45.91.116.12:4225 , piece_id=PQH5MNGKUOET3JPJERGPU2FR4DRMGJVQF2UUEY35QL36XLGH5MOA, seg_pos={0 0}, upload=21459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33092->45.91.116.12:4225: use of closed network connection Sep 7 13:49:32.632586359 (24h21m8.672159479s) [seg_pos={0 0}, upload=21459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:49:32.632709044 (24h21m8.672282164s) [upload=21459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:49:32.632713823 (24h21m8.672286943s) [upload=21459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:49:32.964835227 (24h21m9.004408347s) [upload=21459]: Done uploading segments... Sep 7 13:49:32.964858161 (24h21m9.004431281s) [upload=21459]: Upload finished. err: <nil> Duration: 30.002166967s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.870687154s FAIL (2025-09-07 13:49:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.870756539s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.073669025s FAIL (2025-09-07 13:49:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.073688733s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.........] duration: 29.447209699s FAIL (2025-09-07 13:49:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 29.447234835s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.753388091s FAIL (2025-09-07 13:48:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.753411236s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X........] duration: 13.135550867s FAIL (2025-09-07 13:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 13.135579273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.363308892s FAIL (2025-09-07 13:48:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.363354449s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.......] duration: 29.129043117s FAIL (2025-09-07 13:47:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 29.1290757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.730962718s FAIL (2025-09-07 13:46:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.730986373s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X......] duration: 20.377228462s FAIL (2025-09-07 13:46:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 20.377254872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X.....] duration: 22.465456989s FAIL (2025-09-07 13:46:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 22.465516383s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.050975562s FAIL (2025-09-07 13:45:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.051000519s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X......X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X....] duration: 26.04819151s FAIL (2025-09-07 13:45:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 26.048208343s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.530866898s FAIL (2025-09-07 13:45:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.530898938s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X......X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X...] duration: 16.178956691s FAIL (2025-09-07 13:44:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 16.178992859s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.572071335s FAIL (2025-09-07 13:44:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.572092506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X......X.X.....X.........X....X.XXX.XXX..XX.X.XX..X.XX.X..] duration: 29.430898938s FAIL (2025-09-07 13:44:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 13:41:53.375254237 (24h13m29.414827357s) [upload=21352]: Starting upload Sep 7 13:41:53.375323971 (24h13m29.414897091s) [upload=21352]: Done starting upload Sep 7 13:41:53.375428130 (24h13m29.41500125s) [upload=21352]: Uploading segments... Sep 7 13:41:53.375545916 (24h13m29.415119036s) [upload=21352]: Got next segment. Inline: false Sep 7 13:41:53.375594930 (24h13m29.41516805s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 13:41:53.375600110 (24h13m29.41517323s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 13:41:53.877714339 (24h13m29.917287469s) [seg_pos={0 0}, upload=21352]: Begin upload segment... Sep 7 13:41:53.877764796 (24h13m29.917337956s) [seg_pos={0 0}, upload=21352]: Done begin upload segment. Sep 7 13:41:53.877930122 (24h13m29.917503272s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.877940812 (24h13m29.917513932s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878020405 (24h13m29.917593535s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878101730 (24h13m29.91767485s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878849492 (24h13m29.918422632s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.878954583 (24h13m29.918527733s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.879533340 (24h13m29.919106471s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.882513212 (24h13m29.922086343s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883188404 (24h13m29.922761524s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.883585183 (24h13m29.923158314s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.884057176 (24h13m29.923630306s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.885484087 (24h13m29.925057217s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.891356514 (24h13m29.930929644s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.895902964 (24h13m29.935476094s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.896046559 (24h13m29.935619679s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:53.898144253 (24h13m29.937717373s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Uploading piece... Sep 7 13:41:55.756133769 (24h13m31.795706889s) [upload=21352]: Next returned nil segment Sep 7 13:41:55.756144279 (24h13m31.795717389s) [upload=21352]: Waiting for error group managing segments... Sep 7 13:41:56.058253664 (24h13m32.097826784s) [noise=true , address=77.90.31.19:4842 , piece_id=5USO5GPHQQ3G6JGTAAIP4ZXZTBWSA6XVJRWFGBF6VWKVWE7EM2LA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.100831628 (24h13m32.140404748s) [noise=true , address=45.91.116.11:4118 , piece_id=VK565ZV3QX4XA3STBGC5IE7FVAMF4M64GLYFLA7ABW5IX7IJ6S7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.129158703 (24h13m32.168731823s) [noise=true , address=77.90.31.10:4518 , piece_id=S75XOTJJ72XMQJ5NAIIYKYSQ5KHYPBSAQYQUH7TPQIO3EKXVQ55Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.130193984 (24h13m32.169767104s) [noise=true , address=45.91.116.5:4299 , piece_id=IVYVPZEYPDE3Q6KN4EFMSQF36NRWUCQ7EMNOOA25NXTW2XDHSDQQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.133683370 (24h13m32.17325649s) [noise=true , address=45.91.116.12:4227 , piece_id=TNWFGL5F5HPZ7IF2UZU2SGJFORJ7VQSHQZIUJT6IC65JBBISRDZA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.136224994 (24h13m32.175798114s) [noise=true , address=77.90.31.13:4627 , piece_id=7X357G2BPSKMK4SXKUBRQWSRMWA5QBI42XOGVIQUONPNW3TZ67JQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.141108118 (24h13m32.180681238s) [noise=true , address=45.91.116.13:4154 , piece_id=FR7OC67NE3ADM2OCANDJYUCMPJOY4MBL3IV5UTZ6WX2W2LZ7L6IA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.146613001 (24h13m32.186186121s) [noise=true , address=77.90.31.15:4698 , piece_id=KEFKGPJIXDOTX2R7KITHB32R4VEC7JAHGPTRYPNEPSFM3HGLOOMA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.177943433 (24h13m32.217516553s) [noise=true , address=77.90.31.14:4663 , piece_id=XC7G5EO2IHBR3X4ZN3QAYRNB6TTLCPFHUCOKTXZ5MHW2NIW5WW5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.233755871 (24h13m32.273328992s) [noise=true , address=77.90.31.10:4519 , piece_id=AUYLOLI4DW2QO5Q4HB3KDFT4WE7J3O6HY7EYLUHVBXBTM7GE3F7A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.239017640 (24h13m32.27859076s) [noise=true , address=77.90.31.18:4805 , piece_id=GQMCE2772V4JH346VKQU3W4WYYHQBL2MYFBAYEZGKDEEXJSBDZ2A, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.309311059 (24h13m32.348884179s) [noise=true , address=77.90.31.16:4735 , piece_id=ZZJEES4SQQNP5WOAZ5CFDYD2O3NTW7SGOJPMIXWCP3CDJHOM3M5Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404005670 (24h13m32.44357879s) [noise=true , address=45.91.116.9:4263 , piece_id=IOEPCZRETN4XBILKL42VM7X5BDXTFZXY4GCW4II3SQH6DDS4C5VQ, seg_pos={0 0}, upload=21352]: Done uploading piece. err: <nil> Sep 7 13:41:56.404033132 (24h13m32.443606252s) [seg_pos={0 0}, upload=21352]: Segment reached optimal threshold of 13 pieces. Sep 7 13:41:56.404227475 (24h13m32.443800595s) [noise=true , address=45.91.116.6:4974 , piece_id=3MS7ABIG2IKJX2C3FHOK5NGXOFSXXF6GSVI7X5LQAMKGUZMG7M2Q, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40772->45.91.116.6:4974: use of closed network connection Sep 7 13:41:56.404271388 (24h13m32.443844508s) [noise=true , address=45.91.116.7:4011 , piece_id=MV24RJYPQHRL7CZ75RTG2RQAODKC2NBVGHN5VX4DVQAR65LIPZOA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38138->45.91.116.7:4011: use of closed network connection Sep 7 13:41:56.404278802 (24h13m32.443851922s) [noise=true , address=45.91.116.9:4262 , piece_id=DTQSJFQPCN45UYMUG57HJSPT37CEIF3XVFGIBAHE3I6DFLK76COA, seg_pos={0 0}, upload=21352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48634->45.91.116.9:4262: use of closed network connection Sep 7 13:41:56.404299051 (24h13m32.443872171s) [seg_pos={0 0}, upload=21352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 13:41:56.404361500 (24h13m32.44393462s) [upload=21352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 13:41:56.404365509 (24h13m32.443938629s) [upload=21352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 13:41:56.900088067 (24h13m32.939661177s) [upload=21352]: Done uploading segments... Sep 7 13:41:56.900107264 (24h13m32.939680384s) [upload=21352]: Upload finished. err: <nil> Duration: 29.430918527s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.621865997s FAIL (2025-09-07 13:43:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.621891556s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)