Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X......X............................................] duration: 6.356511198s FAIL (2025-09-08 10:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.356535835s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.323740003s FAIL (2025-09-08 09:59:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.323760723s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 6.132878727s FAIL (2025-09-08 09:59:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.132898805s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.991852586s FAIL (2025-09-08 09:58:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.991875378s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration: 6.010562091s FAIL (2025-09-08 09:58:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.01059295s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.289855735s FAIL (2025-09-08 09:58:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.289879709s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.172276423s FAIL (2025-09-08 09:57:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.172297963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration: 5.921595359s FAIL (2025-09-08 09:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 5.921620076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.177281793s FAIL (2025-09-08 09:56:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.177303254s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.448732384s FAIL (2025-09-08 09:56:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.448753155s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......X........................................] duration: 5.988175614s FAIL (2025-09-08 09:56:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 5.988211823s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.134661005s FAIL (2025-09-08 09:55:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.13468456s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......X.......................................] duration: 5.91976331s FAIL (2025-09-08 09:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 5.919790791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......X......................................] duration: 5.97847289s FAIL (2025-09-08 09:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 5.978498949s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.167588729s FAIL (2025-09-08 09:54:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.167609289s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......X.....................................] duration: 6.028094071s FAIL (2025-09-08 09:54:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.028123076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.251408563s FAIL (2025-09-08 09:53:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.251430184s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......X....................................] duration: 6.13626617s FAIL (2025-09-08 09:53:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.136312197s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.340306779s FAIL (2025-09-08 09:52:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.340329874s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......X...................................] duration: 6.884446011s FAIL (2025-09-08 09:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.884473385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......X..................................] duration: 6.603136358s FAIL (2025-09-08 09:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.603173528s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.496065476s FAIL (2025-09-08 09:52:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.496114951s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.168075282s FAIL (2025-09-08 09:51:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.168091342s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X......X.................................] duration: 6.442293401s FAIL (2025-09-08 09:51:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.442318008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 6.687426961s FAIL (2025-09-08 09:51:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.687510411s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.372143423s FAIL (2025-09-08 09:50:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.372194512s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 6.905724249s FAIL (2025-09-08 09:50:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.905746852s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.2604511s FAIL (2025-09-08 09:49:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.260493682s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 8.15088519s FAIL (2025-09-08 09:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 8.150914095s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.139002055s FAIL (2025-09-08 09:48:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.139018175s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 6.706521742s FAIL (2025-09-08 09:48:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.706550437s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.02720959s FAIL (2025-09-08 09:48:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.027230038s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.232035389s FAIL (2025-09-08 09:47:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.232055157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 6.042962934s FAIL (2025-09-08 09:47:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.042982309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 6.577015876s FAIL (2025-09-08 09:46:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.577078225s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.260233704s FAIL (2025-09-08 09:46:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.260258189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 6.180016419s FAIL (2025-09-08 09:46:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.180042611s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.05724755s FAIL (2025-09-08 09:45:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.05728887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 6.3410418s FAIL (2025-09-08 09:45:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.341068951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 6.68198331s FAIL (2025-09-08 09:45:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.682016543s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.102384042s FAIL (2025-09-08 09:44:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.102411245s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 6.206014313s FAIL (2025-09-08 09:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.206041656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 6.946957404s FAIL (2025-09-08 09:43:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.946987321s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.216013074s FAIL (2025-09-08 09:43:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.21603713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 6.861883875s FAIL (2025-09-08 09:42:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 6.861909073s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.727936045s FAIL (2025-09-08 09:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.727960634s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.304444968s FAIL (2025-09-08 09:42:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.304527296s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......X....................] duration: 5.859376857s FAIL (2025-09-08 09:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 5.859407596s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.048835683s FAIL (2025-09-08 09:41:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.048858025s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......X...................] duration: 9.031459227s FAIL (2025-09-08 09:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 09:25:46.333593945 (43h57m22.373167065s) [upload=38642]: Starting upload Sep 8 09:25:46.333648169 (43h57m22.373221289s) [upload=38642]: Done starting upload Sep 8 09:25:46.333893929 (43h57m22.373467049s) [upload=38642]: Uploading segments... Sep 8 09:25:46.333937984 (43h57m22.373511104s) [upload=38642]: Got next segment. Inline: false Sep 8 09:25:46.333966008 (43h57m22.373539128s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 09:25:46.333970897 (43h57m22.373544017s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 09:25:46.584698149 (43h57m22.624271269s) [seg_pos={0 0}, upload=38642]: Begin upload segment... Sep 8 09:25:46.584754917 (43h57m22.624328047s) [seg_pos={0 0}, upload=38642]: Done begin upload segment. Sep 8 09:25:46.584776489 (43h57m22.624349609s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.584818238 (43h57m22.624391358s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585276355 (43h57m22.624849465s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585353523 (43h57m22.624926643s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.585606326 (43h57m22.625179446s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586043563 (43h57m22.625616683s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586336043 (43h57m22.625909163s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586665193 (43h57m22.626238313s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.586913658 (43h57m22.626486778s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587273256 (43h57m22.626846366s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587549274 (43h57m22.627122394s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.587839700 (43h57m22.62741282s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588166846 (43h57m22.627739966s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588439828 (43h57m22.628012948s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588540491 (43h57m22.628113611s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:46.588831407 (43h57m22.628404527s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Uploading piece... Sep 8 09:25:48.416216108 (43h57m24.455789228s) [upload=38642]: Next returned nil segment Sep 8 09:25:48.416233582 (43h57m24.455806702s) [upload=38642]: Waiting for error group managing segments... Sep 8 09:25:48.706816379 (43h57m24.746389499s) [noise=true , address=77.90.31.15:4697 , piece_id=QH7WX6YZAE7AOD6FACJTSSIDHJ2HSJRE7JHHR7AGB6LM6S3GEEYQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.720064200 (43h57m24.75963732s) [noise=true , address=145.79.227.6:4257 , piece_id=CFRXJIWCJ7SAW2BJF7HXHDIMMJRESPLZUFT2Z7EIBZ2WGVFGSD5A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.731640752 (43h57m24.771213872s) [noise=true , address=77.90.31.18:4807 , piece_id=WZNSGQKIDJ3EAHXROJR74E44YLYUEM2SFS3ZBCJT363FPLBEWINA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.740151257 (43h57m24.779724377s) [noise=true , address=145.79.227.7:4294 , piece_id=47QI6YSPZ5VEKTYWX44A5IP3GFMLQSKYAOMMNKZWJBPMZ4YIXO4A, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.747697568 (43h57m24.787270678s) [noise=true , address=145.79.227.14:4839 , piece_id=L6LVIWKRRYOM7GA6UZK26OMAE7JNEKK36JO65PSJSW3JNZFYZS3Q, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.763932565 (43h57m24.803505675s) [noise=true , address=145.79.227.5:4223 , piece_id=VELFRCJP2CIORCTSNSFFTHJNTIVC4VPPVJRM3JOKBSMNCATM6BFQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.778393115 (43h57m24.817966235s) [noise=true , address=45.91.116.12:4226 , piece_id=WSBGZKI2CKW6LA7F766ULCDZB3V3LT5XFEXYYVPBZ7XVQL26KKMA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.788311766 (43h57m24.827884886s) [noise=true , address=77.90.31.14:4663 , piece_id=5BXBLQ3GWJSV3MKK6BXTVNAAG5VFIPJLQPD32IF4XW6RVRG7GRIQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.835735103 (43h57m24.875308223s) [noise=true , address=77.90.31.10:4518 , piece_id=G267WVK4NVVH4NKMJTVF2F3I6QVXLVJPHH6MD5OOTMZ26CITW6FA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.892038573 (43h57m24.931611693s) [noise=true , address=45.91.116.11:4117 , piece_id=4F5FFHVXYIH246YHZTAM7TR6ECTWIF7EU7UWBLIPHBNLGZSWBDIA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.932303773 (43h57m24.971876893s) [noise=true , address=45.91.116.6:4975 , piece_id=I5M4OQOHSYJ22SYD3Z6LJRMTFPDFKO5RUK6UWB56R6GTRQ34H3WQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:48.954230531 (43h57m24.993803651s) [noise=true , address=145.79.227.5:4222 , piece_id=YN63W66CKWZ4CXDVUEAVW325EKUKOEVXYEH47XILDJDPN2V7CNUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280203539 (43h57m25.319776659s) [noise=true , address=145.79.227.4:4187 , piece_id=V4OBIC37BA2GDA2RNYLDWIQKP6IREMZUCCYAUNPCUTQTUCMDI3VQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: <nil> Sep 8 09:25:49.280229348 (43h57m25.319802468s) [seg_pos={0 0}, upload=38642]: Segment reached optimal threshold of 13 pieces. Sep 8 09:25:49.280345430 (43h57m25.31991855s) [noise=true , address=145.79.227.15:4876 , piece_id=QGV2OOHQ6RDRIQK3JHYALZKZ7GW2IW2K63KOCEWS5WAR2EEUCYCA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280417839 (43h57m25.319990959s) [noise=true , address=145.79.227.17:4971 , piece_id=534NHVVX6YOE43JXC53MV4F2HYMKLPJKNOYMHYFMREY22G54T7NQ, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF Sep 8 09:25:49.280499605 (43h57m25.320072725s) [noise=true , address=45.91.116.7:4010 , piece_id=6DA66OZZ4OCSFGSACVXRXPZY55GPR4NKSWQ6IQL4WUDBVUCDEFUA, seg_pos={0 0}, upload=38642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Sep 8 09:25:49.280518131 (43h57m25.320091251s) [seg_pos={0 0}, upload=38642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 8 09:25:49.280573697 (43h57m25.320146818s) [upload=38642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 09:25:49.280577434 (43h57m25.320150555s) [upload=38642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 09:25:49.635729270 (43h57m25.67530238s) [upload=38642]: Done uploading segments... Sep 8 09:25:49.635747835 (43h57m25.675320955s) [upload=38642]: Upload finished. err: <nil> Duration: 9.031487861s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)