Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X.........................X...............................] duration:
5.871294255s
FAIL (2025-07-09 02:58:56 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.871327077s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [......X..X.....XXXXX...X....X..XXX...X.........X.XXX......X.] duration:
5.547131629s
FAIL (2025-07-09 02:58:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.5471536s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [X......X..X.....XXXXX...X....X..XXX...X.........X.XXX......X] duration:
1m0.000889084s
FAIL (2025-07-09 02:58:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000915535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X......X..........X.........................................] duration:
5.722831734s
FAIL (2025-07-09 02:58:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.722857101s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X....XXX.X.......XX.....XX.X...X.....XXX.........X.X....] duration:
5.951080635s
FAIL (2025-07-09 02:58:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.951113088s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X.........................X..............................] duration:
5.694830352s
FAIL (2025-07-09 02:57:56 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 5.694850112s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X...X.........................X.............................] duration:
9.856502574s
FAIL (2025-07-09 02:57:41 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 9.85653677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX......] duration:
12.02390785s
FAIL (2025-07-09 02:57:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.023918853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.....X....XXX.X.......XX.....XX.X...X.....XXX.........X.X...] duration:
6.130125324s
FAIL (2025-07-09 02:57:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.13014441s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X......X..........X........................................] duration:
5.737239475s
FAIL (2025-07-09 02:57:26 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.737261016s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [X.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX.....] duration:
29.117536931s
FAIL (2025-07-09 02:57:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.117566657s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.X......X..........X.......................................] duration:
5.845892516s
FAIL (2025-07-09 02:56:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.845930609s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.X...X.........................X............................] duration:
33.143968311s
FAIL (2025-07-09 02:56:56 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 33.143995392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.....X....XXX.X.......XX.....XX.X...X.....XXX.........X.X..] duration:
24.797352058s
FAIL (2025-07-09 02:56:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.797372558s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [.X.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX....] duration:
54.806232968s
FAIL (2025-07-09 02:56:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 54.806245552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX.X......X..........X......................................] duration:
5.841916s
FAIL (2025-07-09 02:56:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.84194281s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X.....X....XXX.X.......XX.....XX.X...X.....XXX.........X.X.] duration:
14.418036047s
FAIL (2025-07-09 02:56:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.41812658s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..X...X.........................X...........................] duration:
55.434404008s
FAIL (2025-07-09 02:56:22 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 55.434431322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [XXX.X......X..........X.....................................] duration:
5.774645871s
FAIL (2025-07-09 02:56:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.774674707s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [..X.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX...] duration:
49.802515543s
FAIL (2025-07-09 02:55:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 49.802526052s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X.....X....XXX.X.......XX.....XX.X...X.....XXX.........X.X] duration:
30.001419712s
FAIL (2025-07-09 02:55:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001446024s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.XXX.X......X..........X....................................] duration:
5.873917371s
FAIL (2025-07-09 02:55:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.87394906s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [...X.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX..] duration:
51.125770373s
FAIL (2025-07-09 02:54:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 51.125786432s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X...X.........................X..........................] duration:
24.706615989s
FAIL (2025-07-09 02:54:48 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 24.706631419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..XXX.X......X..........X...................................] duration:
6.019340757s
FAIL (2025-07-09 02:54:26 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.0193681s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X...X.........................X.........................] duration:
56.131546429s
FAIL (2025-07-09 02:54:23 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 56.131566617s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...X.....X....XXX.X.......XX.....XX.X...X.....XXX.........X.] duration:
13.881922772s
FAIL (2025-07-09 02:54:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.881950877s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [....X.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX.] duration:
35.007136206s
FAIL (2025-07-09 02:53:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 35.007230636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X.....X....XXX.X.......XX.....XX.X...X.....XXX.........X] duration:
30.000577028s
FAIL (2025-07-09 02:53:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000773493s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...XXX.X......X..........X..................................] duration:
5.7623676s
FAIL (2025-07-09 02:53:28 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.762392198s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X...X.........................X........................] duration:
23.367724089s
FAIL (2025-07-09 02:53:26 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 23.36775116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [.....X.X......X..X.....XXXXX...X....X..XXX...X.........X.XXX] duration:
1m0.000944148s
FAIL (2025-07-09 02:53:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000954367s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....XXX.X......X..........X.................................] duration:
5.94903833s
FAIL (2025-07-09 02:52:46 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.949064641s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X...X.........................X.......................] duration:
6.352341116s
FAIL (2025-07-09 02:52:36 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 6.352364851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X...X.........................X......................] duration:
17.990987647s
FAIL (2025-07-09 02:52:27 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 17.991015712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....XXX.X......X..........X................................] duration:
6.091237559s
FAIL (2025-07-09 02:52:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.091290399s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.....X.....X....XXX.X.......XX.....XX.X...X.....XXX.........] duration:
14.737341218s
FAIL (2025-07-09 02:52:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.737360524s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........X...X.........................X.....................] duration:
26.228336469s
FAIL (2025-07-09 02:52:04 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 26.228361407s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [......X.X......X..X.....XXXXX...X....X..XXX...X.........X.XX] duration:
1m0.000697324s
FAIL (2025-07-09 02:52:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000709196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [......X.....X....XXX.X.......XX.....XX.X...X.....XXX........] duration:
6.542225983s
FAIL (2025-07-09 02:51:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.542243228s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........X...X.........................X....................] duration:
36.893724788s
FAIL (2025-07-09 02:51:20 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 36.89375195s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......XXX.X......X..........X...............................] duration:
5.846521117s
FAIL (2025-07-09 02:51:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.84654786s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [.......X.X......X..X.....XXXXX...X....X..XXX...X.........X.X] duration:
1m0.001771939s
FAIL (2025-07-09 02:51:02 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001799921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.......X.....X....XXX.X.......XX.....XX.X...X.....XXX.......] duration:
10.814830176s
FAIL (2025-07-09 02:50:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.814852317s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......XXX.X......X..........X..............................] duration:
5.857548005s
FAIL (2025-07-09 02:50:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.857575248s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........X...X.........................X...................] duration:
27.807142983s
FAIL (2025-07-09 02:50:16 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 27.807171608s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [........X.....X....XXX.X.......XX.....XX.X...X.....XXX......] duration:
28.670951059s
FAIL (2025-07-09 02:50:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.670964835s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........X...X.........................X..................] duration:
14.766169729s
FAIL (2025-07-09 02:49:43 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 14.766194185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [........X.X......X..X.....XXXXX...X....X..XXX...X.........X.] duration:
40.896996488s
FAIL (2025-07-09 02:49:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 40.897046985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........XXX.X......X..........X.............................] duration:
5.810829618s
FAIL (2025-07-09 02:49:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.810855558s
core
uplink
us-select-1
|