Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X........................X....] duration: 5.688073685s FAIL (2025-08-21 09:51:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 997c11378ed758e493ce5c2c98e7176a222a3aef5d74a18cafbe295ee3509b43 != before 141660eed5a96ea0dcbf7dd55fd9e8731fd6e78b358ed75391d208ac12f91a56 ======LOGS====== Aug 21 09:48:59.884051773 (304h22m54.315131904s) [upload=316618]: Starting upload Aug 21 09:48:59.884100636 (304h22m54.315180767s) [upload=316618]: Done starting upload Aug 21 09:48:59.884230154 (304h22m54.315310285s) [upload=316618]: Uploading segments... Aug 21 09:48:59.884799824 (304h22m54.315879965s) [upload=316618]: Got next segment. Inline: false Aug 21 09:48:59.885103725 (304h22m54.316183866s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:48:59.885112633 (304h22m54.316192774s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:49:00.331957344 (304h22m54.763037475s) [seg_pos={0 0}, upload=316618]: Begin upload segment... Aug 21 09:49:00.333278321 (304h22m54.764358452s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.335784998 (304h22m54.766865129s) [seg_pos={0 0}, upload=316618]: Done begin upload segment. Aug 21 09:49:00.335807882 (304h22m54.766888003s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336053662 (304h22m54.767133783s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336367612 (304h22m54.767447743s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336675642 (304h22m54.767755773s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337032755 (304h22m54.768112886s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337410187 (304h22m54.768490318s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337618366 (304h22m54.768698497s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337845009 (304h22m54.76892514s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338067345 (304h22m54.769147476s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338353822 (304h22m54.769433953s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338559076 (304h22m54.769639207s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338778055 (304h22m54.769858186s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339006301 (304h22m54.770086422s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339201414 (304h22m54.770281545s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339386699 (304h22m54.77046683s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339629844 (304h22m54.770709975s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339852140 (304h22m54.770932261s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:01.808009614 (304h22m56.239089755s) [upload=316618]: Next returned nil segment Aug 21 09:49:01.808032438 (304h22m56.239112569s) [upload=316618]: Waiting for error group managing segments... Aug 21 09:49:02.181848308 (304h22m56.612928439s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.247077465 (304h22m56.678157606s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.259425001 (304h22m56.690505162s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.303566490 (304h22m56.734646621s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.304374306 (304h22m56.735454437s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.322844957 (304h22m56.753925088s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.323602897 (304h22m56.754683028s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.341580996 (304h22m56.772661127s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.352099842 (304h22m56.783179983s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.354306284 (304h22m56.785386415s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.391345504 (304h22m56.822425646s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.404679337 (304h22m56.835759468s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.479816743 (304h22m56.910896874s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484943462 (304h22m56.916023624s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484978599 (304h22m56.91605874s) [seg_pos={0 0}, upload=316618]: Segment reached optimal threshold of 14 pieces. Aug 21 09:49:02.485140579 (304h22m56.91622073s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485242774 (304h22m56.916322905s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485250600 (304h22m56.916330731s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485405676 (304h22m56.916485827s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection Aug 21 09:49:02.485441195 (304h22m56.916521326s) [seg_pos={0 0}, upload=316618]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 21 09:49:02.485537869 (304h22m56.91661803s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 21 09:49:02.485545664 (304h22m56.916625795s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:49:02.893636886 (304h22m57.324717017s) [upload=316618]: Done uploading segments... Aug 21 09:49:02.893655672 (304h22m57.324735803s) [upload=316618]: Upload finished. err: <nil> Duration: 5.688102019s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...X...X...................X...............X.XX.............] duration: 1.976720435s FAIL (2025-08-21 09:51:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.976744701s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X........................X...] duration: 5.837413649s FAIL (2025-08-21 09:51:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 997c11378ed758e493ce5c2c98e7176a222a3aef5d74a18cafbe295ee3509b43 != before 141660eed5a96ea0dcbf7dd55fd9e8731fd6e78b358ed75391d208ac12f91a56 ======LOGS====== Aug 21 09:48:59.884051773 (304h22m54.315131904s) [upload=316618]: Starting upload Aug 21 09:48:59.884100636 (304h22m54.315180767s) [upload=316618]: Done starting upload Aug 21 09:48:59.884230154 (304h22m54.315310285s) [upload=316618]: Uploading segments... Aug 21 09:48:59.884799824 (304h22m54.315879965s) [upload=316618]: Got next segment. Inline: false Aug 21 09:48:59.885103725 (304h22m54.316183866s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:48:59.885112633 (304h22m54.316192774s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:49:00.331957344 (304h22m54.763037475s) [seg_pos={0 0}, upload=316618]: Begin upload segment... Aug 21 09:49:00.333278321 (304h22m54.764358452s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.335784998 (304h22m54.766865129s) [seg_pos={0 0}, upload=316618]: Done begin upload segment. Aug 21 09:49:00.335807882 (304h22m54.766888003s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336053662 (304h22m54.767133783s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336367612 (304h22m54.767447743s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336675642 (304h22m54.767755773s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337032755 (304h22m54.768112886s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337410187 (304h22m54.768490318s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337618366 (304h22m54.768698497s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337845009 (304h22m54.76892514s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338067345 (304h22m54.769147476s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338353822 (304h22m54.769433953s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338559076 (304h22m54.769639207s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338778055 (304h22m54.769858186s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339006301 (304h22m54.770086422s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339201414 (304h22m54.770281545s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339386699 (304h22m54.77046683s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339629844 (304h22m54.770709975s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339852140 (304h22m54.770932261s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:01.808009614 (304h22m56.239089755s) [upload=316618]: Next returned nil segment Aug 21 09:49:01.808032438 (304h22m56.239112569s) [upload=316618]: Waiting for error group managing segments... Aug 21 09:49:02.181848308 (304h22m56.612928439s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.247077465 (304h22m56.678157606s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.259425001 (304h22m56.690505162s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.303566490 (304h22m56.734646621s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.304374306 (304h22m56.735454437s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.322844957 (304h22m56.753925088s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.323602897 (304h22m56.754683028s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.341580996 (304h22m56.772661127s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.352099842 (304h22m56.783179983s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.354306284 (304h22m56.785386415s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.391345504 (304h22m56.822425646s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.404679337 (304h22m56.835759468s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.479816743 (304h22m56.910896874s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484943462 (304h22m56.916023624s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484978599 (304h22m56.91605874s) [seg_pos={0 0}, upload=316618]: Segment reached optimal threshold of 14 pieces. Aug 21 09:49:02.485140579 (304h22m56.91622073s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485242774 (304h22m56.916322905s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485250600 (304h22m56.916330731s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485405676 (304h22m56.916485827s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection Aug 21 09:49:02.485441195 (304h22m56.916521326s) [seg_pos={0 0}, upload=316618]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 21 09:49:02.485537869 (304h22m56.91661803s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 21 09:49:02.485545664 (304h22m56.916625795s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:49:02.893636886 (304h22m57.324717017s) [upload=316618]: Done uploading segments... Aug 21 09:49:02.893655672 (304h22m57.324735803s) [upload=316618]: Upload finished. err: <nil> Duration: 5.83744003s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 3.898814162s FAIL (2025-08-21 09:50:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.898845342s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 3.960310606s FAIL (2025-08-21 09:50:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.960336337s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [....X...X...................X...............X.XX............] duration: 1.792800129s FAIL (2025-08-21 09:50:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.79282793s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X........................X..] duration: 5.64198437s FAIL (2025-08-21 09:50:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 997c11378ed758e493ce5c2c98e7176a222a3aef5d74a18cafbe295ee3509b43 != before 141660eed5a96ea0dcbf7dd55fd9e8731fd6e78b358ed75391d208ac12f91a56 ======LOGS====== Aug 21 09:48:59.884051773 (304h22m54.315131904s) [upload=316618]: Starting upload Aug 21 09:48:59.884100636 (304h22m54.315180767s) [upload=316618]: Done starting upload Aug 21 09:48:59.884230154 (304h22m54.315310285s) [upload=316618]: Uploading segments... Aug 21 09:48:59.884799824 (304h22m54.315879965s) [upload=316618]: Got next segment. Inline: false Aug 21 09:48:59.885103725 (304h22m54.316183866s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:48:59.885112633 (304h22m54.316192774s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:49:00.331957344 (304h22m54.763037475s) [seg_pos={0 0}, upload=316618]: Begin upload segment... Aug 21 09:49:00.333278321 (304h22m54.764358452s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.335784998 (304h22m54.766865129s) [seg_pos={0 0}, upload=316618]: Done begin upload segment. Aug 21 09:49:00.335807882 (304h22m54.766888003s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336053662 (304h22m54.767133783s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336367612 (304h22m54.767447743s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336675642 (304h22m54.767755773s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337032755 (304h22m54.768112886s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337410187 (304h22m54.768490318s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337618366 (304h22m54.768698497s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337845009 (304h22m54.76892514s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338067345 (304h22m54.769147476s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338353822 (304h22m54.769433953s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338559076 (304h22m54.769639207s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338778055 (304h22m54.769858186s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339006301 (304h22m54.770086422s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339201414 (304h22m54.770281545s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339386699 (304h22m54.77046683s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339629844 (304h22m54.770709975s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339852140 (304h22m54.770932261s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:01.808009614 (304h22m56.239089755s) [upload=316618]: Next returned nil segment Aug 21 09:49:01.808032438 (304h22m56.239112569s) [upload=316618]: Waiting for error group managing segments... Aug 21 09:49:02.181848308 (304h22m56.612928439s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.247077465 (304h22m56.678157606s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.259425001 (304h22m56.690505162s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.303566490 (304h22m56.734646621s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.304374306 (304h22m56.735454437s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.322844957 (304h22m56.753925088s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.323602897 (304h22m56.754683028s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.341580996 (304h22m56.772661127s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.352099842 (304h22m56.783179983s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.354306284 (304h22m56.785386415s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.391345504 (304h22m56.822425646s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.404679337 (304h22m56.835759468s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.479816743 (304h22m56.910896874s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484943462 (304h22m56.916023624s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484978599 (304h22m56.91605874s) [seg_pos={0 0}, upload=316618]: Segment reached optimal threshold of 14 pieces. Aug 21 09:49:02.485140579 (304h22m56.91622073s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485242774 (304h22m56.916322905s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485250600 (304h22m56.916330731s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485405676 (304h22m56.916485827s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection Aug 21 09:49:02.485441195 (304h22m56.916521326s) [seg_pos={0 0}, upload=316618]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 21 09:49:02.485537869 (304h22m56.91661803s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 21 09:49:02.485545664 (304h22m56.916625795s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:49:02.893636886 (304h22m57.324717017s) [upload=316618]: Done uploading segments... Aug 21 09:49:02.893655672 (304h22m57.324735803s) [upload=316618]: Upload finished. err: <nil> Duration: 5.642017212s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 3.891520046s FAIL (2025-08-21 09:49:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.891551986s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X........................X.] duration: 5.763708294s FAIL (2025-08-21 09:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 997c11378ed758e493ce5c2c98e7176a222a3aef5d74a18cafbe295ee3509b43 != before 141660eed5a96ea0dcbf7dd55fd9e8731fd6e78b358ed75391d208ac12f91a56 ======LOGS====== Aug 21 09:48:59.884051773 (304h22m54.315131904s) [upload=316618]: Starting upload Aug 21 09:48:59.884100636 (304h22m54.315180767s) [upload=316618]: Done starting upload Aug 21 09:48:59.884230154 (304h22m54.315310285s) [upload=316618]: Uploading segments... Aug 21 09:48:59.884799824 (304h22m54.315879965s) [upload=316618]: Got next segment. Inline: false Aug 21 09:48:59.885103725 (304h22m54.316183866s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:48:59.885112633 (304h22m54.316192774s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:49:00.331957344 (304h22m54.763037475s) [seg_pos={0 0}, upload=316618]: Begin upload segment... Aug 21 09:49:00.333278321 (304h22m54.764358452s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.335784998 (304h22m54.766865129s) [seg_pos={0 0}, upload=316618]: Done begin upload segment. Aug 21 09:49:00.335807882 (304h22m54.766888003s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336053662 (304h22m54.767133783s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336367612 (304h22m54.767447743s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336675642 (304h22m54.767755773s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337032755 (304h22m54.768112886s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337410187 (304h22m54.768490318s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337618366 (304h22m54.768698497s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337845009 (304h22m54.76892514s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338067345 (304h22m54.769147476s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338353822 (304h22m54.769433953s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338559076 (304h22m54.769639207s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338778055 (304h22m54.769858186s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339006301 (304h22m54.770086422s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339201414 (304h22m54.770281545s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339386699 (304h22m54.77046683s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339629844 (304h22m54.770709975s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339852140 (304h22m54.770932261s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:01.808009614 (304h22m56.239089755s) [upload=316618]: Next returned nil segment Aug 21 09:49:01.808032438 (304h22m56.239112569s) [upload=316618]: Waiting for error group managing segments... Aug 21 09:49:02.181848308 (304h22m56.612928439s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.247077465 (304h22m56.678157606s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.259425001 (304h22m56.690505162s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.303566490 (304h22m56.734646621s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.304374306 (304h22m56.735454437s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.322844957 (304h22m56.753925088s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.323602897 (304h22m56.754683028s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.341580996 (304h22m56.772661127s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.352099842 (304h22m56.783179983s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.354306284 (304h22m56.785386415s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.391345504 (304h22m56.822425646s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.404679337 (304h22m56.835759468s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.479816743 (304h22m56.910896874s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484943462 (304h22m56.916023624s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484978599 (304h22m56.91605874s) [seg_pos={0 0}, upload=316618]: Segment reached optimal threshold of 14 pieces. Aug 21 09:49:02.485140579 (304h22m56.91622073s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485242774 (304h22m56.916322905s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485250600 (304h22m56.916330731s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485405676 (304h22m56.916485827s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection Aug 21 09:49:02.485441195 (304h22m56.916521326s) [seg_pos={0 0}, upload=316618]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 21 09:49:02.485537869 (304h22m56.91661803s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 21 09:49:02.485545664 (304h22m56.916625795s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:49:02.893636886 (304h22m57.324717017s) [upload=316618]: Done uploading segments... Aug 21 09:49:02.893655672 (304h22m57.324735803s) [upload=316618]: Upload finished. err: <nil> Duration: 5.763738551s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....X...X...................X...............X.XX...........] duration: 1.800152646s FAIL (2025-08-21 09:49:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.800177383s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 4.051430645s FAIL (2025-08-21 09:49:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.051460902s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X........................X] duration: 5.961898472s FAIL (2025-08-21 09:49:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 997c11378ed758e493ce5c2c98e7176a222a3aef5d74a18cafbe295ee3509b43 != before 141660eed5a96ea0dcbf7dd55fd9e8731fd6e78b358ed75391d208ac12f91a56 ======LOGS====== Aug 21 09:48:59.884051773 (304h22m54.315131904s) [upload=316618]: Starting upload Aug 21 09:48:59.884100636 (304h22m54.315180767s) [upload=316618]: Done starting upload Aug 21 09:48:59.884230154 (304h22m54.315310285s) [upload=316618]: Uploading segments... Aug 21 09:48:59.884799824 (304h22m54.315879965s) [upload=316618]: Got next segment. Inline: false Aug 21 09:48:59.885103725 (304h22m54.316183866s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:48:59.885112633 (304h22m54.316192774s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:49:00.331957344 (304h22m54.763037475s) [seg_pos={0 0}, upload=316618]: Begin upload segment... Aug 21 09:49:00.333278321 (304h22m54.764358452s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.335784998 (304h22m54.766865129s) [seg_pos={0 0}, upload=316618]: Done begin upload segment. Aug 21 09:49:00.335807882 (304h22m54.766888003s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336053662 (304h22m54.767133783s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336367612 (304h22m54.767447743s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.336675642 (304h22m54.767755773s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337032755 (304h22m54.768112886s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337410187 (304h22m54.768490318s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337618366 (304h22m54.768698497s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.337845009 (304h22m54.76892514s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338067345 (304h22m54.769147476s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338353822 (304h22m54.769433953s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338559076 (304h22m54.769639207s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.338778055 (304h22m54.769858186s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339006301 (304h22m54.770086422s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339201414 (304h22m54.770281545s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339386699 (304h22m54.77046683s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339629844 (304h22m54.770709975s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:00.339852140 (304h22m54.770932261s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Uploading piece... Aug 21 09:49:01.808009614 (304h22m56.239089755s) [upload=316618]: Next returned nil segment Aug 21 09:49:01.808032438 (304h22m56.239112569s) [upload=316618]: Waiting for error group managing segments... Aug 21 09:49:02.181848308 (304h22m56.612928439s) [noise=true , address=79.127.220.97:4391 , piece_id=JQRFQJFW276FEQ6RCRSHTOX3BBAE776W75ZU5N6TCLLFYAGUACSA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.247077465 (304h22m56.678157606s) [noise=true , address=45.91.116.13:4175 , piece_id=ZP7GNOUXMQ4OWKBT4WMEQTLRUBPSNL3LTOBYNHYLRC4BJPTQTCNQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.259425001 (304h22m56.690505162s) [noise=true , address=104.36.82.15:4262 , piece_id=DXTWAQY756FMY4JMU3CHM3MYLCV6PAPOLTUR7BEP4VDRJJASAHRA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.303566490 (304h22m56.734646621s) [noise=true , address=77.90.31.17:4782 , piece_id=GVW57JPTTN3BZN7V6YUVEJLBJIHGPLXKMALYDNSL3D2JCPZFIKFQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.304374306 (304h22m56.735454437s) [noise=true , address=79.127.219.48:4106 , piece_id=CL2NFO3NQZNEW2PJ7D5P6LZINIEV2VKFODUECGSZ4BXHKBVFCDGA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.322844957 (304h22m56.753925088s) [noise=true , address=77.90.31.15:4726 , piece_id=M4UWU4PSUX72NT74E3Y62RRDYJQBJCSWEDWMCENQHXCU7FTC2DZA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.323602897 (304h22m56.754683028s) [noise=true , address=45.91.116.11:4136 , piece_id=Y375OWXJ2XSKUD6VSRKXASIO2DWCA2SDWZ7V3LVR627OUNQX3K2A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.341580996 (304h22m56.772661127s) [noise=true , address=45.91.116.9:4287 , piece_id=CCPDCHWOSNZKKRF2JGKTYYFNCDP5E75F4SPHTYSS46EEMBTJSMQA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.352099842 (304h22m56.783179983s) [noise=true , address=79.127.219.51:4208 , piece_id=FY46KZLX7BYD7RHEDOA36R7HNIKMV4ILTXN5MU3UM4YN4Q2WI47A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.354306284 (304h22m56.785386415s) [noise=true , address=77.90.31.10:4544 , piece_id=TNLJFADRMSIJEKVGYGKXU5QVM2UMWEYQNAGRXNU5QZCWP4B2SBHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.391345504 (304h22m56.822425646s) [noise=true , address=77.90.31.16:4739 , piece_id=GQKK2MUSU7YYVKCJPQTVZNJN6OZTJ6X5VVTRBI5WJE32WWDN6MPA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.404679337 (304h22m56.835759468s) [noise=true , address=77.90.31.21:4888 , piece_id=XOVSQGGQYALSPXTB4VSR254L2TPKLIMFB2KTBS5FHLCT7YPBZ7BQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.479816743 (304h22m56.910896874s) [noise=true , address=77.90.31.19:4851 , piece_id=XYCGYRSGQQCY2CSS4IJBINXTLZNHLOJZZH63GR6FCHQH53MU72YA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484943462 (304h22m56.916023624s) [noise=true , address=77.90.31.13:4660 , piece_id=MEUSR3KXE6TGP2IYA7FK5NGLRP2GB6XZ43KDILHCRHYNDFMVNJKA, seg_pos={0 0}, upload=316618]: Done uploading piece. err: <nil> Aug 21 09:49:02.484978599 (304h22m56.91605874s) [seg_pos={0 0}, upload=316618]: Segment reached optimal threshold of 14 pieces. Aug 21 09:49:02.485140579 (304h22m56.91622073s) [noise=true , address=104.36.82.23:4461 , piece_id=YQVU6LNOVXQYD5H2UVFN3SSXQIH5PVPGUDX5HVVYXLH6P5NLFK4Q, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485242774 (304h22m56.916322905s) [noise=true , address=77.90.31.11:4563 , piece_id=IYW3MZYUBZFHP7ZP7EWDJEMBQ36FQTIIPAN7LGAIE7DJTISBAD4A, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485250600 (304h22m56.916330731s) [noise=true , address=104.36.82.23:4480 , piece_id=MTZRMW2QOB3ETFXL7LMXOEMLHNIMQVXE5QHBMU56OF2GXWZOVOHQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU): protocol: expected piece hash; context canceled; EOF Aug 21 09:49:02.485405676 (304h22m56.916485827s) [noise=true , address=104.36.82.51:4049 , piece_id=IZDHDHLCRLZD3RN75KMFYVUAN337267Z62F2ARKHHUOYROWGEAEQ, seg_pos={0 0}, upload=316618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60046->104.36.82.51:4049: use of closed network connection Aug 21 09:49:02.485441195 (304h22m56.916521326s) [seg_pos={0 0}, upload=316618]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 21 09:49:02.485537869 (304h22m56.91661803s) [upload=316618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 21 09:49:02.485545664 (304h22m56.916625795s) [upload=316618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:49:02.893636886 (304h22m57.324717017s) [upload=316618]: Done uploading segments... Aug 21 09:49:02.893655672 (304h22m57.324735803s) [upload=316618]: Upload finished. err: <nil> Duration: 5.961930463s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [......X...X...................X...............X.XX..........] duration: 1.819775581s FAIL (2025-08-21 09:49:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.819805037s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 3.93120144s FAIL (2025-08-21 09:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.931230547s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.877484622s FAIL (2025-08-21 09:48:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.877519379s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 3.896202195s FAIL (2025-08-21 09:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.896227764s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......X...X...................X...............X.XX.........] duration: 1.845373569s FAIL (2025-08-21 09:47:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.84539504s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.584053155s FAIL (2025-08-21 09:47:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.584080068s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.581391217s FAIL (2025-08-21 09:47:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.58141866s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 3.841994633s FAIL (2025-08-21 09:47:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.842024049s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [........X...X...................X...............X.XX........] duration: 1.820891212s FAIL (2025-08-21 09:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.820926649s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.77964704s FAIL (2025-08-21 09:46:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.779679061s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 3.902386608s FAIL (2025-08-21 09:46:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.902414954s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 3.90436793s FAIL (2025-08-21 09:46:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.904405371s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.........X...X...................X...............X.XX.......] duration: 2.041094267s FAIL (2025-08-21 09:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.041115687s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 3.968056096s FAIL (2025-08-21 09:45:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.968090711s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......................................X....................] duration: 5.728954582s FAIL (2025-08-21 09:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.728974992s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [..........X...X...................X...............X.XX......] duration: 2.006699147s FAIL (2025-08-21 09:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.006733472s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 3.821802769s FAIL (2025-08-21 09:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.821823359s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...........X...X...................X...............X.XX.....] duration: 1.820176699s FAIL (2025-08-21 09:44:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.820210244s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......................................X...................] duration: 5.59515154s FAIL (2025-08-21 09:44:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.595201296s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X......................................X..................] duration: 5.671136359s FAIL (2025-08-21 09:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.671159332s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 3.901636213s FAIL (2025-08-21 09:43:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.901664959s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [............X...X...................X...............X.XX....] duration: 1.807263802s FAIL (2025-08-21 09:43:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.807289692s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X......................................X.................] duration: 5.822407797s FAIL (2025-08-21 09:43:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.822434687s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X......................................X................] duration: 5.518558819s FAIL (2025-08-21 09:43:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.518586172s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 3.905592738s FAIL (2025-08-21 09:43:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.905620272s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.............X...X...................X...............X.XX...] duration: 1.796660957s FAIL (2025-08-21 09:42:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.796688459s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.X......................................X...............] duration: 5.881398226s FAIL (2025-08-21 09:42:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.881423364s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 3.87104972s FAIL (2025-08-21 09:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.871074256s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.X......................................X..............] duration: 5.659512726s FAIL (2025-08-21 09:41:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.659539947s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [..............X...X...................X...............X.XX..] duration: 2.018473813s FAIL (2025-08-21 09:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.018501657s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...............X...X...................X...............X.XX.] duration: 1.844345157s FAIL (2025-08-21 09:41:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.844378362s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.X......................................X.............] duration: 5.510887386s FAIL (2025-08-21 09:40:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.510910922s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 3.906268896s FAIL (2025-08-21 09:40:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.906292149s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.X......................................X............] duration: 5.699560709s FAIL (2025-08-21 09:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.699586216s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [................X...X...................X...............X.XX] duration: 865.846039ms FAIL (2025-08-21 09:40:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:40:40.014562937 (304h14m34.445643069s) [upload=316472]: Starting upload Aug 21 09:40:40.014609146 (304h14m34.445689267s) [upload=316472]: Done starting upload Aug 21 09:40:40.014660875 (304h14m34.445740996s) [upload=316472]: Uploading segments... Aug 21 09:40:40.014693287 (304h14m34.445773418s) [upload=316472]: Got next segment. Inline: true Aug 21 09:40:40.014696714 (304h14m34.445776845s) [upload=316472]: Waiting for error group managing segments... Aug 21 09:40:40.014730849 (304h14m34.44581097s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:40.014734796 (304h14m34.445814927s) [upload=316472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:40.014737332 (304h14m34.445817463s) [upload=316472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:40.230140739 (304h14m34.66122087s) [upload=316472]: Done uploading segments... Aug 21 09:40:40.230158053 (304h14m34.661238184s) [upload=316472]: Upload finished. err: metaclient: Too Many Requests Duration:* 865.870576ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.................X...X...................X...............X.X] duration: 2.101228241s FAIL (2025-08-21 09:40:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 8e597dd5cd80a42d161282a3c9d1335433f9b44bc8fbb78918cc374833520e25 != before 9108710b7268f0e1516683e5b28d2afcc5c98e1e0e9041266c030149dad35b96 ======LOGS====== Aug 21 09:40:24.941519663 (304h14m19.372599794s) [upload=316468]: Starting upload Aug 21 09:40:24.941592092 (304h14m19.372672223s) [upload=316468]: Done starting upload Aug 21 09:40:24.941641416 (304h14m19.372721547s) [upload=316468]: Uploading segments... Aug 21 09:40:24.941679258 (304h14m19.372759389s) [upload=316468]: Got next segment. Inline: true Aug 21 09:40:24.941683656 (304h14m19.372763787s) [upload=316468]: Waiting for error group managing segments... Aug 21 09:40:24.941716210 (304h14m19.372796341s) [upload=316468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:40:24.941719876 (304h14m19.372800007s) [upload=316468]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 21 09:40:24.941723754 (304h14m19.372803885s) [upload=316468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 21 09:40:25.688244539 (304h14m20.11932467s) [upload=316468]: Done uploading segments... Aug 21 09:40:25.688265239 (304h14m20.11934537s) [upload=316468]: Upload finished. err: <nil> Duration:* 2.101251424s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.X......................................X...........] duration: 5.956200379s FAIL (2025-08-21 09:40:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 21 09:31:01.185465443 (304h4m55.616545574s) [upload=316302]: Starting upload Aug 21 09:31:01.185526440 (304h4m55.616606571s) [upload=316302]: Done starting upload Aug 21 09:31:01.185637422 (304h4m55.616717553s) [upload=316302]: Uploading segments... Aug 21 09:31:01.185789724 (304h4m55.616869855s) [upload=316302]: Got next segment. Inline: false Aug 21 09:31:01.185906557 (304h4m55.616986688s) [upload=316302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 21 09:31:01.185912920 (304h4m55.616993081s) [upload=316302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 21 09:31:01.406186667 (304h4m55.837266798s) [upload=316302]: Next segment err: metaclient: Too Many Requests Aug 21 09:31:01.406210583 (304h4m55.837290704s) [upload=316302]: Done uploading segments... Aug 21 09:31:01.406236192 (304h4m55.837316323s) [upload=316302]: Upload finished. err: metaclient: Too Many Requests Duration: 5.956226799s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 3.906919538s FAIL (2025-08-21 09:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Uploading piece... Aug 21 09:25:01.622689552 (303h58m56.053769693s) [upload=316197]: Next returned nil segment Aug 21 09:25:01.622711935 (303h58m56.053792066s) [upload=316197]: Waiting for error group managing segments... Aug 21 09:25:02.006851367 (303h58m56.437931508s) [noise=true , address=81.171.14.70:4048 , piece_id=6WA7PH7Z3OMPLVS6YT2V5Y65RCAOUO2FZS2NFBVSPCILGGHEGG7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.009968412 (303h58m56.441048553s) [noise=true , address=81.171.14.70:4041 , piece_id=NI6YRNGFIQYXUFI7IO5IIYE73ROQIDYXXZVX2NZEXZP5HP5TWFYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.011808102 (303h58m56.442888243s) [noise=true , address=81.171.14.70:4042 , piece_id=TFDZVXYNFMYRNLHT5UH2TRODZOELQSU2S5PF4BCMVIUEEYJEYJVQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.013448941 (303h58m56.444529092s) [noise=true , address=81.171.14.70:4032 , piece_id=ZSBGQZHT64XDRRLCFGZ3PTI4AV6RYZNMBJSSWWTCQH7JCOTJT4SA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031414296 (303h58m56.462494437s) [noise=true , address=89.149.201.147:4022 , piece_id=LHIXBNFSNQ3CNITGGK5WAANLASDPMK6XTAXDPZ5MYRB56DQPSJ4A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.031706434 (303h58m56.462786575s) [noise=true , address=81.171.14.70:4045 , piece_id=5BLSQ7JM7M5XZXOIIS4LNMOQEOP5HC5PKDTALI2LDKXMHIQAFBTQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.041720265 (303h58m56.472800406s) [noise=true , address=89.149.201.147:4020 , piece_id=WTPUOJEEEH5I5PCBIGHQGUYBEAFATZSVDDY2CRQ4LNH7JLA5PQRQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.047790370 (303h58m56.478870511s) [noise=true , address=89.149.201.147:4019 , piece_id=YH4HVEETKLEPWA35GS6N7GXISKYNFBWZ3AFLBHPYUXCSQARBCO5Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.048412099 (303h58m56.47949224s) [noise=true , address=89.149.201.147:4024 , piece_id=QEXK3EVLQM4XKGE325MMW2H3IH3V3KVWTVU2JSMXZ723FLVJMOWQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.052927248 (303h58m56.484007389s) [noise=true , address=81.171.14.70:4028 , piece_id=XSYK2OOKRXMPXNXU7FXMOXX5PVM5FZOHYTY5NGSSVEYDXEU46HPA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.065188018 (303h58m56.496268159s) [noise=true , address=89.149.201.147:4011 , piece_id=ZGHLOUWBWGG6QBW7POKIGV6SEU2UAWMYYI4GCYSUAAQCM3SSJZ7A, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.083998499 (303h58m56.51507864s) [noise=true , address=89.149.201.147:4007 , piece_id=GTNAWJKCSKD7FK4GJ5HP3QPTGZN27HA4QEL66VVEV3ZC2F3TIYJA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084577036 (303h58m56.515657177s) [noise=true , address=89.149.201.147:4008 , piece_id=UFD6KMTZ4RYAI3ZGKQAOP2UWHIS7UVZPNQOAUN3DFOE7GRVD6HEA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.084693098 (303h58m56.515773229s) [noise=true , address=89.149.201.147:4012 , piece_id=7CUBW7VMY4Q23IHB3FQ76AFVAY3LKWQRNGPWMIQJWFEOLSVYMRYQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:02.152607593 (303h58m56.583687725s) [noise=true , address=81.171.14.70:4047 , piece_id=7LERJRDEST2KNDNXCIBTYSIA7MUH6KQ4ITQICWHCMRSRLQ2HYYOA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: <nil> Aug 21 09:25:59.970209376 (303h59m54.401289497s) [noise=true , address=185.167.204.206:4043 , piece_id=IYMQCVK2KUMBH3IXZHNUZX3IQZ4LOZDWIGIFKZHPJKFSSEVSUITQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: rpc: dial tcp 185.167.204.206:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970242529 (303h59m54.40132266s) [noise=true , address=185.167.204.206:4060 , piece_id=OVW3PJKFJU7DDEPEHJ2NI7YB2XMSQ34LSXJHRN7MBBUF6TI3TCYA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12RJC6wKVpcVGVNxVSXEYVXCVhC1vDPjbduZ1cFmCDPaYf1WTSr, address:185.167.204.206:4060): protocol: rpc: dial tcp 185.167.204.206:4060: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970257307 (303h59m54.401337438s) [noise=true , address=185.167.204.206:4071 , piece_id=DJRQVWGFKKPLSZIPFXFDTAB6UQJ2CXEEFQG7B4ZTFAPULLQLUHFQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: rpc: dial tcp 185.167.204.206:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970285360 (303h59m54.401365491s) [noise=true , address=185.167.204.206:4041 , piece_id=UQ356DF727DLQCBMPPBWRUX7VU4V5S3WGW36DF2XCKFSKGB7XMNA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970290601 (303h59m54.401370732s) [noise=true , address=185.167.204.206:4040 , piece_id=QMKNXSFLJ775HE6C2YXJPTCXDVLYLIXC5YUVED463MMKQQFDOZKA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: write tcp 188.166.225.170:33230->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970315699 (303h59m54.40139582s) [noise=true , address=185.167.204.205:4015 , piece_id=AGHBBB4XLTYMXOYRAEWLZPGZIGYZLWFAHIIMJCEYEGPDCCAHQMDA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: rpc: dial tcp 185.167.204.205:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970319857 (303h59m54.401399988s) [noise=true , address=185.167.204.205:4036 , piece_id=NRZHG2H5L4BZN6SZERNRSOXIAS2FHKUQJL4TMZO4MNRGPJYWEQ4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: rpc: dial tcp 185.167.204.205:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970347871 (303h59m54.401428002s) [noise=true , address=185.167.204.205:4012 , piece_id=RK7U25EJSWCALVQBDRXB5BSG5CQW3DA3M42VPMHVOXBU6OLQJYLA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: rpc: dial tcp 185.167.204.205:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970373490 (303h59m54.401453621s) [noise=true , address=185.167.204.206:4063 , piece_id=JN2WHWF4M74DB5FQYXDA2FGRPQVNFPQRNI4UAJRLN6AWXXYH3PZQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: rpc: dial tcp 185.167.204.206:4063: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970379561 (303h59m54.401459692s) [noise=true , address=185.167.204.205:4024 , piece_id=PW2BKAX6H6PG76A7WR3Z6L5IL73WEO44MQWVQ4PE6Y6CKO4TXN4Q, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: rpc: dial tcp 185.167.204.205:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970385182 (303h59m54.401465313s) [noise=true , address=185.167.204.206:4049 , piece_id=64H54XD6CXWGJMUDETU2VEMH7Y6VCNSZUMNRX6RINMKI4TYNE2GA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: rpc: dial tcp 185.167.204.206:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970392215 (303h59m54.401472346s) [noise=true , address=185.167.204.206:4059 , piece_id=ZVPH7MXNJO4RRWNRHIWZ74VZFWSYQKUAW3RDLRSJQM2DBHUGPGAQ, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970407885 (303h59m54.401488016s) [noise=true , address=185.167.204.205:4003 , piece_id=AFNLL2L3AICH2AOUVG6OPIBD4FFMINXCO4CTAEQ4FYA4565DK7LA, seg_pos={0 0}, upload=316197]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:51828->185.167.204.205:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 21 09:25:59.970428695 (303h59m54.401508826s) [seg_pos={0 0}, upload=316197]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Aug 21 09:26:00.138485232 (303h59m54.569565363s) [upload=316197]: Done uploading segments... Aug 21 09:26:00.138516251 (303h59m54.569596382s) [upload=316197]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 3.906946601s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)