Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 330747 reports (Page 6331 of 6615)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X.....X....XXX.X.......XX.....XX.X...X.....XXX.....] duration: 13.210659332s FAIL (2025-07-09 02:49:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.210678248s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X...X.........................X.................] duration: 37.372180573s FAIL (2025-07-09 02:49:03 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 37.372195571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X........X.X......X..X.....XXXXX...X....X..XXX...X.........X] duration: 1m0.001017619s FAIL (2025-07-09 02:48:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001041974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X...X.........................X................] duration: 41.660866405s FAIL (2025-07-09 02:48:25 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 41.660890962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XXX.X......X..........X............................] duration: 5.87183024s FAIL (2025-07-09 02:48:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871861299s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..........X.....X....XXX.X.......XX.....XX.X...X.....XXX....] duration: 6.64960742s FAIL (2025-07-09 02:48:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.649644261s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........X.....X....XXX.X.......XX.....XX.X...X.....XXX...] duration: 26.994930562s FAIL (2025-07-09 02:48:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.994947112s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XX........X.X......X..X.....XXXXX...X....X..XXX...X.........] duration: 48.994725958s FAIL (2025-07-09 02:47:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.994739674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [............X.....X....XXX.X.......XX.....XX.X...X.....XXX..] duration: 17.750506508s FAIL (2025-07-09 02:47:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.750532869s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.........XXX.X......X..........X...........................] duration: 14.061317369s FAIL (2025-07-09 02:47:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.061345302s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............X.....X....XXX.X.......XX.....XX.X...X.....XXX.] duration: 9.607010709s FAIL (2025-07-09 02:47:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.607022312s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.XX........X.X......X..X.....XXXXX...X....X..XXX...X........] duration: 46.933222292s FAIL (2025-07-09 02:47:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.933247962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X...X.........................X...............] duration: 29.209439453s FAIL (2025-07-09 02:46:59 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 29.209457718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............X.....X....XXX.X.......XX.....XX.X...X.....XXX] duration: 30.001411332s FAIL (2025-07-09 02:46:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001434475s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.........XXX.X......X..........X..........................] duration: 5.822602853s FAIL (2025-07-09 02:46:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822631266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X...X.........................X..............] duration: 36.593388412s FAIL (2025-07-09 02:46:30 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 36.59341813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............X.....X....XXX.X.......XX.....XX.X...X.....XX] duration: 30.000598896s FAIL (2025-07-09 02:45:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000619166s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.........XXX.X......X..........X.........................] duration: 5.806345555s FAIL (2025-07-09 02:45:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.806368971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X.XX........X.X......X..X.....XXXXX...X....X..XXX...X.......] duration: 39.189301999s FAIL (2025-07-09 02:45:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.189322377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...............X.....X....XXX.X.......XX.....XX.X...X.....X] duration: 30.000822058s FAIL (2025-07-09 02:45:20 UTC) orbiter warning: context deadline exceeded Duration: 30.000844349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X...............X...X.........................X.............] duration: 26.094897042s FAIL (2025-07-09 02:45:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 26.094923223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX........X.X......X..X.....XXXXX...X....X..XXX...X......] duration: 44.466187365s FAIL (2025-07-09 02:45:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.466211181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.........XXX.X......X..........X........................] duration: 6.02774561s FAIL (2025-07-09 02:44:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.027773394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X...............X...X.........................X............] duration: 41.496322331s FAIL (2025-07-09 02:44:48 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 41.496348652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...............X.....X....XXX.X.......XX.....XX.X...X.....] duration: 7.796312943s FAIL (2025-07-09 02:44:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.796336169s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X...............X.....X....XXX.X.......XX.....XX.X...X....] duration: 21.386644627s FAIL (2025-07-09 02:44:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.386665248s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..X.XX........X.X......X..X.....XXXXX...X....X..XXX...X.....] duration: 37.65593451s FAIL (2025-07-09 02:44:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.655978726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.........XXX.X......X..........X.......................] duration: 5.822813703s FAIL (2025-07-09 02:44:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822841916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X...............X...X.........................X...........] duration: 43.307677888s FAIL (2025-07-09 02:44:00 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 43.307692086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.........XXX.X......X..........X......................] duration: 5.867145586s FAIL (2025-07-09 02:43:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.867170213s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...............X.....X....XXX.X.......XX.....XX.X...X...] duration: 16.389536406s FAIL (2025-07-09 02:43:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.389560154s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...X.XX........X.X......X..X.....XXXXX...X....X..XXX...X....] duration: 27.56856932s FAIL (2025-07-09 02:43:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.568631539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X...............X...X.........................X..........] duration: 42.721982239s FAIL (2025-07-09 02:43:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 42.722008159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X...X.XX........X.X......X..X.....XXXXX...X....X..XXX...X...] duration: 39.034005308s FAIL (2025-07-09 02:42:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.034024856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.........XXX.X......X..........X.....................] duration: 7.371503371s FAIL (2025-07-09 02:42:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.371528539s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X...............X.....X....XXX.X.......XX.....XX.X...X..] duration: 6.446045095s FAIL (2025-07-09 02:42:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.446065535s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...............X...X.........................X.........] duration: 35.59826545s FAIL (2025-07-09 02:42:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 35.598283355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X......X.........XXX.X......X..........X....................] duration: 5.906183651s FAIL (2025-07-09 02:42:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90620917s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X...............X.....X....XXX.X.......XX.....XX.X...X.] duration: 8.006327407s FAIL (2025-07-09 02:41:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.006350561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...............X...X.........................X........] duration: 34.816429312s FAIL (2025-07-09 02:41:36 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 34.816458639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX...X.XX........X.X......X..X.....XXXXX...X....X..XXX...X..] duration: 37.029857516s FAIL (2025-07-09 02:41:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.029878597s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X...X...............X.....X....XXX.X.......XX.....XX.X...X] duration: 30.001195022s FAIL (2025-07-09 02:41:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001239858s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...............X...X.........................X.......] duration: 16.672828037s FAIL (2025-07-09 02:40:58 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 16.672855349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X......X.........XXX.X......X..........X...................] duration: 5.830696132s FAIL (2025-07-09 02:40:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.830729506s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X......X.........XXX.X......X..........X..................] duration: 24.927959694s FAIL (2025-07-09 02:40:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 24.928005642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX...X.XX........X.X......X..X.....XXXXX...X....X..XXX...X.] duration: 47.993212601s FAIL (2025-07-09 02:40:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.994029585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...............X...X.........................X......] duration: 58.027949013s FAIL (2025-07-09 02:40:31 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 58.027981144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X...X...............X.....X....XXX.X.......XX.....XX.X...] duration: 13.971469834s FAIL (2025-07-09 02:39:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.971493249s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XX...X.XX........X.X......X..X.....XXXXX...X....X..XXX...X] duration: 1m0.001178499s FAIL (2025-07-09 02:39:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001189841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X......X.........XXX.X......X..........X.................] duration: 5.780785337s FAIL (2025-07-09 02:39:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 377h15m47.862153541s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments... Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil> Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments... Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment... Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment. Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece... Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments... Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil> Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments... Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780812389s core uplink us-select-1

1
...
6329
6330
6331
6332
6333
...
6615