Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.532024203s
FAIL (2025-07-09 03:07:27 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 6.532048841s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [XX.X.......XX.....XX.X...X.....XXX.........X.X..............] duration:
6.450582775s
FAIL (2025-07-09 03:07:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.450604976s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [.....XXXXX...X....X..XXX...X.........X.XXX......X...........] duration:
6.929552102s
FAIL (2025-07-09 03:07:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.929574937s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X..............................................X.......] duration:
5.77383299s
FAIL (2025-07-09 03:06:53 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.773852036s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
8.016865392s
FAIL (2025-07-09 03:06:26 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 8.016890589s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [X.....XXXXX...X....X..XXX...X.........X.XXX......X..........] duration:
5.475114946s
FAIL (2025-07-09 03:06:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.475139132s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XXX.X.......XX.....XX.X...X.....XXX.........X.X.............] duration:
6.186136257s
FAIL (2025-07-09 03:06:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.186155764s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X..............................................X......] duration:
5.770049944s
FAIL (2025-07-09 03:06:06 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.770112915s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [.X.....XXXXX...X....X..XXX...X.........X.XXX......X.........] duration:
5.58887706s
FAIL (2025-07-09 03:05:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.58890279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.802850088s
FAIL (2025-07-09 03:05:46 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.802878011s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X..............................................X.....] duration:
5.820489332s
FAIL (2025-07-09 03:05:27 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.820515441s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.XXX.X.......XX.....XX.X...X.....XXX.........X.X............] duration:
6.530958286s
FAIL (2025-07-09 03:05:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.530985458s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.858330447s
FAIL (2025-07-09 03:05:07 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.858356118s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [..X.....XXXXX...X....X..XXX...X.........X.XXX......X........] duration:
5.520650037s
FAIL (2025-07-09 03:04:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.520675926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X..............................................X....] duration:
5.744332675s
FAIL (2025-07-09 03:04:50 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.744356409s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
7.052115554s
FAIL (2025-07-09 03:04:45 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 7.052158054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..XXX.X.......XX.....XX.X...X.....XXX.........X.X...........] duration:
5.979751273s
FAIL (2025-07-09 03:04:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.979771122s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X..X.....XXXXX...X....X..XXX...X.........X.XXX......X.......] duration:
6.371573563s
FAIL (2025-07-09 03:04:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.371594424s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X..............................................X...] duration:
5.781947615s
FAIL (2025-07-09 03:03:48 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.781971181s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
8.280687131s
FAIL (2025-07-09 03:03:44 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 8.280711318s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...XXX.X.......XX.....XX.X...X.....XXX.........X.X..........] duration:
5.981219988s
FAIL (2025-07-09 03:03:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.981243523s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X..............................................X..] duration:
5.871211707s
FAIL (2025-07-09 03:03:25 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.871248106s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X..........X..............................................X.] duration:
5.75303209s
FAIL (2025-07-09 03:03:09 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 5.753048311s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X..........X..............................................X] duration:
30.001045197s
FAIL (2025-07-09 03:03:03 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
expected piece hash; context canceled; EOF
Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments...
Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil>
Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload
Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload
Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments...
Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false
Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment...
Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment.
Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece...
Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment
Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments...
Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil>
Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces.
Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments...
Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil>
Duration: 30.001064003s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.X..X.....XXXXX...X....X..XXX...X.........X.XXX......X......] duration:
5.448651105s
FAIL (2025-07-09 03:03:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.448677115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..X..X.....XXXXX...X....X..XXX...X.........X.XXX......X.....] duration:
5.582467021s
FAIL (2025-07-09 03:02:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.582488504s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [....XXX.X.......XX.....XX.X...X.....XXX.........X.X.........] duration:
5.931023693s
FAIL (2025-07-09 03:02:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.931049892s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.696262602s
FAIL (2025-07-09 03:02:30 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.696292498s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X....XXX.X.......XX.....XX.X...X.....XXX.........X.X........] duration:
5.973305946s
FAIL (2025-07-09 03:01:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.973327167s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X..........X..............................................] duration:
5.748299257s
FAIL (2025-07-09 03:01:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.748324565s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...X..X.....XXXXX...X....X..XXX...X.........X.XXX......X....] duration:
5.450645759s
FAIL (2025-07-09 03:01:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.450662132s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.01974395s
FAIL (2025-07-09 03:01:20 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 6.019765381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X....XXX.X.......XX.....XX.X...X.....XXX.........X.X.......] duration:
6.038538654s
FAIL (2025-07-09 03:01:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.038598079s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.904589401s
FAIL (2025-07-09 03:00:47 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.904625941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X..........X.............................................] duration:
5.754255735s
FAIL (2025-07-09 03:00:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.754285192s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [....X..X.....XXXXX...X....X..XXX...X.........X.XXX......X...] duration:
5.481216545s
FAIL (2025-07-09 03:00:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.48124015s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.....X..X.....XXXXX...X....X..XXX...X.........X.XXX......X..] duration:
5.871329262s
FAIL (2025-07-09 03:00:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.871354829s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X..........X............................................] duration:
5.679882134s
FAIL (2025-07-09 03:00:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.679901962s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X.........................X.................................] duration:
5.545960564s
FAIL (2025-07-09 02:59:54 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.545986404s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..X....XXX.X.......XX.....XX.X...X.....XXX.........X.X......] duration:
6.131014555s
FAIL (2025-07-09 02:59:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.131036286s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...X....XXX.X.......XX.....XX.X...X.....XXX.........X.X.....] duration:
23.671232885s
FAIL (2025-07-09 02:59:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.671255597s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X..........X...........................................] duration:
5.809450037s
FAIL (2025-07-09 02:59:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.80947796s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X.........................X................................] duration:
5.890871288s
FAIL (2025-07-09 02:59:24 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.890899732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X..........X..........................................] duration:
6.200684505s
FAIL (2025-07-09 02:59:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.2007081s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X.........................X...............................] duration:
5.871294255s
FAIL (2025-07-09 02:58:56 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.871327077s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [......X..X.....XXXXX...X....X..XXX...X.........X.XXX......X.] duration:
5.547131629s
FAIL (2025-07-09 02:58:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.5471536s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [X......X..X.....XXXXX...X....X..XXX...X.........X.XXX......X] duration:
1m0.000889084s
FAIL (2025-07-09 02:58:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000915535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X......X..........X.........................................] duration:
5.722831734s
FAIL (2025-07-09 02:58:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.722857101s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X....XXX.X.......XX.....XX.X...X.....XXX.........X.X....] duration:
5.951080635s
FAIL (2025-07-09 02:58:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.951113088s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X.........................X..............................] duration:
5.694830352s
FAIL (2025-07-09 02:57:56 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.694850112s
core
uplink
us-select-1
us1-placement-20
|