Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [.....XX...X.XX........X.X......X..X.....XXXXX...X....X..XXX.] duration:
52.555764529s
FAIL (2025-07-09 02:36:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 52.555786092s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X.X.X.X...X...............X.....X....XXX.X.......XX.....XX] duration:
30.000826009s
FAIL (2025-07-09 02:36:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000849354s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X......X.........XXX.X......X..........X.............] duration:
5.890457725s
FAIL (2025-07-09 02:35:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.890487523s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [............X...............X...X.........................X.] duration:
33.423943304s
FAIL (2025-07-09 02:35:52 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 33.423961319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X......X.........XXX.X......X..........X............] duration:
20.265879916s
FAIL (2025-07-09 02:35:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 20.265900716s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
20 out of 60 is failed (threshold=3) [X.....XX...X.XX........X.X......X..X.....XXXXX...X....X..XXX] duration:
1m0.000453129s
FAIL (2025-07-09 02:35:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000476392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.............X...............X...X.........................X] duration:
1m0.002130869s
FAIL (2025-07-09 02:35:18 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
t next segment. Inline: false
Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment...
Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment.
Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece...
Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment
Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments...
Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil>
Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF
Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection
Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments...
Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil>
Duration: 1m0.002147039s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X.X.X.X...X...............X.....X....XXX.X.......XX.....X] duration:
30.001339705s
FAIL (2025-07-09 02:35:16 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00136402s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........X......X.........XXX.X......X..........X...........] duration:
5.805504085s
FAIL (2025-07-09 02:34:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.805526388s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........X......X.........XXX.X......X..........X..........] duration:
17.52390607s
FAIL (2025-07-09 02:34:21 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 17.523966728s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
20 out of 60 is failed (threshold=3) [XX.....XX...X.XX........X.X......X..X.....XXXXX...X....X..XX] duration:
1m0.00173076s
FAIL (2025-07-09 02:34:15 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001753825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X.............X...............X...X.........................] duration:
31.794152043s
FAIL (2025-07-09 02:34:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 31.794183654s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X...X.X.X.X...X...............X.....X....XXX.X.......XX.....] duration:
9.060073732s
FAIL (2025-07-09 02:34:03 UTC)
orbiter warning: context deadline exceeded
Duration: 9.060095605s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........X......X.........XXX.X......X..........X.........] duration:
5.749687439s
FAIL (2025-07-09 02:33:37 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.749713298s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X...X.X.X.X...X...............X.....X....XXX.X.......XX....] duration:
26.618331026s
FAIL (2025-07-09 02:33:28 UTC)
orbiter warning: context deadline exceeded
Duration: 26.618352657s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [.XX.....XX...X.XX........X.X......X..X.....XXXXX...X....X..X] duration:
1m0.000845643s
FAIL (2025-07-09 02:33:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000861412s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X.............X...............X...X........................] duration:
20.625952667s
FAIL (2025-07-09 02:33:00 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 20.625971332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [............X......X.........XXX.X......X..........X........] duration:
5.763535059s
FAIL (2025-07-09 02:32:58 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.763559145s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............X......X.........XXX.X......X..........X.......] duration:
5.817132595s
FAIL (2025-07-09 02:32:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.817153786s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X.............X...............X...X.......................] duration:
21.249844069s
FAIL (2025-07-09 02:32:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 21.249869437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X.............X...............X...X......................] duration:
43.244995601s
FAIL (2025-07-09 02:32:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 43.245031539s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..XX.....XX...X.XX........X.X......X..X.....XXXXX...X....X..] duration:
55.802815973s
FAIL (2025-07-09 02:32:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 55.802831212s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X...X.X.X.X...X...............X.....X....XXX.X.......XX...] duration:
9.86716124s
FAIL (2025-07-09 02:32:05 UTC)
orbiter warning: context deadline exceeded
Duration: 9.867179355s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..............X......X.........XXX.X......X..........X......] duration:
5.73564893s
FAIL (2025-07-09 02:31:37 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.735688115s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X.............X...............X...X.....................] duration:
33.640149632s
FAIL (2025-07-09 02:31:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 33.64017396s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...XX.....XX...X.XX........X.X......X..X.....XXXXX...X....X.] duration:
35.448780446s
FAIL (2025-07-09 02:31:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 35.448794362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X..X...X.X.X.X...X...............X.....X....XXX.X.......XX..] duration:
6.108307135s
FAIL (2025-07-09 02:30:53 UTC)
orbiter warning: context deadline exceeded
Duration: 6.108342041s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [....XX.....XX...X.XX........X.X......X..X.....XXXXX...X....X] duration:
1m0.00092737s
FAIL (2025-07-09 02:30:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000960895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...............X......X.........XXX.X......X..........X.....] duration:
5.72983047s
FAIL (2025-07-09 02:30:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.729862891s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X..X...X.X.X.X...X...............X.....X....XXX.X.......XX.] duration:
19.837871528s
FAIL (2025-07-09 02:30:29 UTC)
orbiter warning: context deadline exceeded
Duration: 19.837910693s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [................X......X.........XXX.X......X..........X....] duration:
5.753474784s
FAIL (2025-07-09 02:30:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.753502076s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X..X...X.X.X.X...X...............X.....X....XXX.X.......XX] duration:
30.00051887s
FAIL (2025-07-09 02:30:09 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000532617s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X.............X...............X...X....................] duration:
28.078164215s
FAIL (2025-07-09 02:30:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 28.078184334s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X..X...X.X.X.X...X...............X.....X....XXX.X.......X] duration:
30.001047562s
FAIL (2025-07-09 02:29:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001080895s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.................X......X.........XXX.X......X..........X...] duration:
5.854787027s
FAIL (2025-07-09 02:29:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.854817874s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......X.............X...............X...X...................] duration:
49.28097423s
FAIL (2025-07-09 02:29:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 49.280998396s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X....XX.....XX...X.XX........X.X......X..X.....XXXXX...X....] duration:
36.603272783s
FAIL (2025-07-09 02:29:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 36.603282742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....X..X...X.X.X.X...X...............X.....X....XXX.X.......] duration:
16.018777344s
FAIL (2025-07-09 02:28:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.018798484s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [XX....XX.....XX...X.XX........X.X......X..X.....XXXXX...X...] duration:
31.588041973s
FAIL (2025-07-09 02:28:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 31.588053405s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......X.............X...............X...X..................] duration:
32.202006013s
FAIL (2025-07-09 02:28:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 32.202031222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [X.................X......X.........XXX.X......X..........X..] duration:
7.03484886s
FAIL (2025-07-09 02:28:37 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.034898084s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X.................X......X.........XXX.X......X..........X.] duration:
24.387774139s
FAIL (2025-07-09 02:28:28 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 24.387797843s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.....X..X...X.X.X.X...X...............X.....X....XXX.X......] duration:
9.657595004s
FAIL (2025-07-09 02:28:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.657611235s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
20 out of 60 is failed (threshold=3) [XXX....XX.....XX...X.XX........X.X......X..X.....XXXXX...X..] duration:
55.151772661s
FAIL (2025-07-09 02:28:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 55.151785034s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........X.............X...............X...X.................] duration:
44.031676902s
FAIL (2025-07-09 02:27:46 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 44.03169654s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..X.................X......X.........XXX.X......X..........X] duration:
30.225023164s
FAIL (2025-07-09 02:27:23 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
377h15m47.862153541s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:13.091516664 (377h15m47.862157989s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:13.385504369 (377h15m48.156145704s) [upload=301826]: Done uploading segments...
Jul 9 02:26:13.385522533 (377h15m48.156163868s) [upload=301826]: Upload finished. err: <nil>
Jul 9 02:26:54.016479548 (377h16m28.787120873s) [upload=301835]: Starting upload
Jul 9 02:26:54.016549942 (377h16m28.787191277s) [upload=301835]: Done starting upload
Jul 9 02:26:54.016635355 (377h16m28.78727669s) [upload=301835]: Uploading segments...
Jul 9 02:26:54.016797706 (377h16m28.787439041s) [upload=301835]: Got next segment. Inline: false
Jul 9 02:26:54.016823676 (377h16m28.787465011s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:26:54.016828054 (377h16m28.787469389s) [upload=301835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:26:54.242995408 (377h16m29.013636743s) [seg_pos={0 0}, upload=301835]: Begin upload segment...
Jul 9 02:26:54.243038040 (377h16m29.013679375s) [seg_pos={0 0}, upload=301835]: Done begin upload segment.
Jul 9 02:26:54.243057076 (377h16m29.013698412s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243095359 (377h16m29.013736684s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243418738 (377h16m29.014060073s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243499513 (377h16m29.014140848s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243739010 (377h16m29.014380345s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.243860073 (377h16m29.014501408s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244022824 (377h16m29.014664159s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244121913 (377h16m29.014763238s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244270388 (377h16m29.014911723s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244512390 (377h16m29.015153725s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244744144 (377h16m29.015385479s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.244959707 (377h16m29.015601042s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245160080 (377h16m29.015801425s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245382275 (377h16m29.0160236s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245635569 (377h16m29.016276924s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:54.245987833 (377h16m29.016629168s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Uploading piece...
Jul 9 02:26:55.905028368 (377h16m30.675669703s) [upload=301835]: Next returned nil segment
Jul 9 02:26:55.905044229 (377h16m30.675685564s) [upload=301835]: Waiting for error group managing segments...
Jul 9 02:26:56.243483897 (377h16m31.014125282s) [noise=true , address=104.36.82.16:4295 , piece_id=DL4ZHYDXITM2HSEVRYMQINUYELZKM5SAT7YIHPY3LZW2YEPI2SDQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.310059506 (377h16m31.080700841s) [noise=true , address=45.91.116.12:4225 , piece_id=XFF5G46NIAUOO2PEZCWPJWLT4RUL4TD4C6OLMOWY3S7ICUBEPE3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329432059 (377h16m31.100073394s) [noise=true , address=77.90.31.18:4806 , piece_id=R2PRRBSSBESVZHNZTTLMKURTBF72BHVDCZCGWK35GEPJT5SMQSUQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.329484579 (377h16m31.100125914s) [noise=true , address=77.90.31.16:4733 , piece_id=WCG4BTP5RO75KTRQVNFLCNIYPEMO35V6P3ECDAYCW4GXQFWXYJVQ, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.332195677 (377h16m31.102837012s) [noise=true , address=77.90.31.17:4772 , piece_id=S6PAWXV4RSEHTF2WLP7I55S77TFKBE5JEU35AW55GEO6FSH3Q4UA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.374976272 (377h16m31.145617607s) [noise=true , address=45.91.116.10:4082 , piece_id=RRKG4RF6TFZN4NCSAKHSV6P5WNRC7CWUBVESSDZY22XTAAJ3XCFA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.388748190 (377h16m31.159389525s) [noise=true , address=45.91.116.7:4010 , piece_id=BUZGBYS3AULOI5YT3EZAJ3RVIJIRA35BLPPSM3YPSFOS6EGPUW4Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.453390932 (377h16m31.224032267s) [noise=true , address=45.91.116.13:4155 , piece_id=GRDCXIFC6BYSBCVFZ5G55EPHGHDKBWPFCYZ7RMKC6OPU4OYICK3A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.458939747 (377h16m31.229581072s) [noise=true , address=45.91.116.11:4119 , piece_id=RSBE2D264WDEM2F5KJ3QBTUU5C4KOFQWP27N7D5JVPMIU2KGP72A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.596790687 (377h16m31.367432023s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMPYBZ6SDHVYKHCZQ3R4ENZQJZGDBMU3DZ2JAWHXOPODFTX45A, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:26:56.902778310 (377h16m31.673419635s) [noise=true , address=45.91.116.6:4973 , piece_id=LFCLEPIJPSE7F6HXE7AIP6BMXA7DJSD5OB2GJ6EDDKKLG534BVWA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:20.143259362 (377h16m54.913900697s) [noise=true , address=104.36.82.2:4729 , piece_id=JYGLDQHNSNP5ZHPPXPSCTI7KOGLJITJJIN5QGXK3TQW4VFHYOP7Q, seg_pos={0 0}, upload=301835]: Done uploading piece. err: <nil>
Jul 9 02:27:23.383573880 (377h16m58.154215205s) [noise=true , address=104.36.82.14:4222 , piece_id=Z6YIK536BY522AZ27LNZ2RXINXKU57MKWKNL222QO5PSVJHFRBBA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:27:23.383710842 (377h16m58.154352177s) [noise=true , address=104.36.82.11:4114 , piece_id=NMREYAIFR32EXGI55JV3CLIU6JFOWTDEBMSUWUZTRJJVNGRBYPQA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->104.36.82.11:4114: use of closed network connection
Jul 9 02:27:23.383747772 (377h16m58.154389107s) [noise=true , address=104.36.82.2:4731 , piece_id=YYWJUGQJNVRCT2DTXBZMVIB36U2TQTHUD2IUM5KG5QFB3WCKPGIA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42754->104.36.82.2:4731: use of closed network connection
Jul 9 02:27:23.383791997 (377h16m58.154433322s) [noise=true , address=104.36.82.8:4972 , piece_id=G5B5JPZCAEMZLBNC5QXYKCPQ4QU2JWXZLBO3EDBWCLIPE7XT2ATA, seg_pos={0 0}, upload=301835]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48434->104.36.82.8:4972: use of closed network connection
Jul 9 02:27:23.383851331 (377h16m58.154492666s) [seg_pos={0 0}, upload=301835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:27:23.606532524 (377h16m58.377173859s) [upload=301835]: Done uploading segments...
Jul 9 02:27:23.606592630 (377h16m58.377233965s) [upload=301835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.22504724s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [......X..X...X.X.X.X...X...............X.....X....XXX.X.....] duration:
6.134559524s
FAIL (2025-07-09 02:27:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.134581795s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
20 out of 60 is failed (threshold=3) [.XXX....XX.....XX...X.XX........X.X......X..X.....XXXXX...X.] duration:
55.403412446s
FAIL (2025-07-09 02:27:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 55.40342535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........X.............X...............X...X................] duration:
57.725477633s
FAIL (2025-07-09 02:27:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
.
Jul 9 02:12:07.627735816 (377h1m42.398377151s) [upload=301626]: Got next segment. Inline: false
Jul 9 02:12:07.627768629 (377h1m42.398409964s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:12:07.627773568 (377h1m42.398414903s) [upload=301626]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:12:08.075105689 (377h1m42.845747024s) [seg_pos={0 0}, upload=301626]: Begin upload segment...
Jul 9 02:12:08.075172738 (377h1m42.845814073s) [seg_pos={0 0}, upload=301626]: Done begin upload segment.
Jul 9 02:12:08.075200341 (377h1m42.845841676s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075226711 (377h1m42.845868046s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.075642747 (377h1m42.846284102s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076099570 (377h1m42.846740905s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076193791 (377h1m42.846835126s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076453428 (377h1m42.847094763s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076686334 (377h1m42.847327669s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076926883 (377h1m42.847568218s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.076956250 (377h1m42.847597585s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077181110 (377h1m42.847822445s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077198733 (377h1m42.847840068s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077433213 (377h1m42.848074548s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077452068 (377h1m42.848093403s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077655618 (377h1m42.848296953s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.077884866 (377h1m42.848526201s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078100028 (377h1m42.848741363s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078310551 (377h1m42.848951886s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:08.078506095 (377h1m42.84914743s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Uploading piece...
Jul 9 02:12:09.631539904 (377h1m44.402181239s) [upload=301626]: Next returned nil segment
Jul 9 02:12:09.631556275 (377h1m44.4021976s) [upload=301626]: Waiting for error group managing segments...
Jul 9 02:12:10.026556908 (377h1m44.797198243s) [noise=true , address=77.90.31.19:4871 , piece_id=UCM2KSWROOFOIGLTBJFY4NF3FKOORX5GRHCBA7DOCAO5HPFYSNAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.029581435 (377h1m44.80022277s) [noise=true , address=77.90.31.18:4836 , piece_id=YDR5T7QRVZGJDCYG27O2ZKQI2Z6BRHSU5WMJXNVCLVAVMNXZOWNA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.030348031 (377h1m44.800989366s) [noise=true , address=77.90.31.13:4634 , piece_id=UHAWAVHQOYTKA63UJ3RGNECZJ3GVTM3I5ERHJZRSKVBU75J5TJAQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.037300182 (377h1m44.807941517s) [noise=true , address=77.90.31.12:4617 , piece_id=UXULKHKF4ZYNM7JRYS2RDWMDUU2ZLH5HA43AVM5KBPUPXW5UYRGQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.038590761 (377h1m44.809232096s) [noise=true , address=77.90.31.14:4675 , piece_id=KFBY3NAIKPH4PRQUTVKJYDQ7ZBY2YHBR4VA2GM4MNNUDZC2NXBZA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.040614022 (377h1m44.811255347s) [noise=true , address=77.90.31.17:4799 , piece_id=OAL6NZCSUMUSLJ4JKBBD54YXL35LJ5BBUZQKEEQPBLQQ4XFNWXYQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.243165042 (377h1m45.013806377s) [noise=true , address=45.91.116.8:4049 , piece_id=SM6AYMUGJRZUEAD6Y4PJM2QDR4QXPOUOXEKLVVTILGYWJZL2D6OQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:10.294115297 (377h1m45.064756632s) [noise=true , address=45.91.116.4:4964 , piece_id=SAYQHQB75OGS2JVQER7JKXKY4OT6VGPCYBHW66Z5T4LFYMVV5GNQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:12.752127764 (377h1m47.522769099s) [noise=true , address=45.91.116.12:4240 , piece_id=3YYFNOWZIDOFNS3RDQE7OJ6XBHRMTSVKGEN2WINWIUDPPXE7LIQQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:29.819338082 (377h2m4.589979417s) [noise=true , address=104.36.82.60:4345 , piece_id=HVM7MDSDGCQCTI2CNMHLDECV6O34BXH65ZJNLXBQCEOXYDIZ45SA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:35.349456003 (377h2m10.120097338s) [noise=true , address=104.36.82.2:4757 , piece_id=YDH5KCYKYK3LOGSFPVTBNIX6YNZZE2TA6JDUNYV2A6UU27GTS2XA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:40.626537094 (377h2m15.397178419s) [noise=true , address=104.36.82.44:4872 , piece_id=24BE6IRU5NKQQVEJYFD4VD5Q46D64IESMJOTQ4ZSEH5AFRBY6EFQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:12:42.766847833 (377h2m17.537489168s) [noise=true , address=104.36.82.20:4459 , piece_id=W2XL4XXRCPTQ5E7H4YOBS3WYQDTJ7U7XXQAFWIWFKEATO6TTYDHQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: <nil>
Jul 9 02:13:06.981429964 (377h2m41.752071299s) [noise=true , address=104.36.82.7:4944 , piece_id=EJUAYL4NEBXYNGFWMUXQNGPGWL27FKLB66S23M5QWZADSXDG7CEQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981472496 (377h2m41.752113821s) [noise=true , address=104.36.82.36:4628 , piece_id=N22NBPASRYZCYQ7BSTVYLXI5XO7VJJ2QK5XGHPIKJREH7YUA67NA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; write tcp 188.166.225.170:43488->104.36.82.36:4628: use of closed network connection; EOF
Jul 9 02:13:06.981498496 (377h2m41.752139821s) [noise=true , address=104.36.82.18:4375 , piece_id=N4DM2NJ7FKRODQBDUMDRRHH3ZCQGHJPOGXDQ3UD5P3PGTSJZURJA, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981567808 (377h2m41.752209143s) [noise=true , address=104.36.82.23:4474 , piece_id=TOCDRGPZ734SJKANQ6NA3RMHZHTYD3WLSY4KWCR5IBK3WTSP5L4A, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:13:06.981595070 (377h2m41.752236395s) [noise=true , address=104.36.82.5:4861 , piece_id=QQIXGQBY42UYJ5OZPLPG4MVFKZZT3HHCYTX6DPEZZJSMXQV7T2IQ, seg_pos={0 0}, upload=301626]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53458->104.36.82.5:4861: use of closed network connection
Jul 9 02:13:06.981611983 (377h2m41.752253318s) [seg_pos={0 0}, upload=301626]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:13:07.214001409 (377h2m41.984642744s) [upload=301626]: Done uploading segments...
Jul 9 02:13:07.214031306 (377h2m41.984672641s) [upload=301626]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 57.7254922s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X......X..X...X.X.X.X...X...............X.....X....XXX.X....] duration:
23.345188551s
FAIL (2025-07-09 02:26:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.345208348s
edge
gateway
us-select-1
|