Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.710167174s FAIL (2025-08-16 16:05:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.710188443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [....X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X] duration: 12.151910566s FAIL (2025-08-16 16:05:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 16 16:05:27.916122149 (190h39m22.34720228s) [upload=198338]: Starting upload Aug 16 16:05:27.916169229 (190h39m22.34724936s) [upload=198338]: Done starting upload Aug 16 16:05:27.916311171 (190h39m22.347391302s) [upload=198338]: Uploading segments... Aug 16 16:05:27.916393368 (190h39m22.347473489s) [upload=198338]: Got next segment. Inline: false Aug 16 16:05:27.916432212 (190h39m22.347512333s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:05:27.916436451 (190h39m22.347516582s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 16:05:34.330059713 (190h39m28.761139904s) [seg_pos={0 0}, upload=198338]: Begin upload segment... Aug 16 16:05:34.330142301 (190h39m28.761222432s) [seg_pos={0 0}, upload=198338]: Done begin upload segment. Aug 16 16:05:34.330182148 (190h39m28.761262289s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330200312 (190h39m28.761280453s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330926893 (190h39m28.762007024s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.330942242 (190h39m28.762022373s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331525619 (190h39m28.76260575s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331531049 (190h39m28.76261118s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331942917 (190h39m28.763023058s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.331957344 (190h39m28.763037475s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332406254 (190h39m28.763486405s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332521564 (190h39m28.763601705s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.332774107 (190h39m28.763854248s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333105160 (190h39m28.764185291s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333429822 (190h39m28.764509963s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.333710268 (190h39m28.764790399s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334103742 (190h39m28.765183873s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334579692 (190h39m28.765659833s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.334870068 (190h39m28.765950199s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:34.335273469 (190h39m28.7663536s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Uploading piece... Aug 16 16:05:35.811636174 (190h39m30.242716305s) [upload=198338]: Next returned nil segment Aug 16 16:05:35.811656744 (190h39m30.242736875s) [upload=198338]: Waiting for error group managing segments... Aug 16 16:05:36.213433670 (190h39m30.644513801s) [noise=true , address=79.127.220.97:4381 , piece_id=SPIPLA323FBOCBRPM2I2LTIR5AIZOPQLDAITPFRGMX2CYNBN7UFA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.272010409 (190h39m30.70309054s) [noise=true , address=104.36.82.5:4849 , piece_id=OGENDA4KRAKSFS5YI6WEYCD4P6BTNP7YR7Y2YBFEZPA53VN6JRLQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.328834736 (190h39m30.759914867s) [noise=true , address=104.36.82.9:4025 , piece_id=T73AIQRP6XKBW74ZBEIVN7JKAZ2BZCWQLGY7VGCTQCOLPS3FZ4VA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.335734207 (190h39m30.766814328s) [noise=true , address=77.90.31.11:4581 , piece_id=SIXD6XJDCVF6IWQRYNV2HSQQBSSB5N4T4JWWX5K74KPEF3HI4O6Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.348504793 (190h39m30.779584924s) [noise=true , address=45.91.116.13:4167 , piece_id=LVQQOHAENCGBDRNQHHYCAWCGHDTAAJZDZTZUKN5XAOHXIHE5NVXA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.375100119 (190h39m30.80618025s) [noise=true , address=104.36.82.14:4230 , piece_id=6P45V6YNJNCGJEADHA7FBKQIIPMLK77MT4TXWI66UMPVQHIS264A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.380335747 (190h39m30.811415878s) [noise=true , address=45.91.116.13:4176 , piece_id=VSJ2NUQOXRZIH4HWFJXBHL6EYMCPMGEFCT2PCKKCRGY4JXSSE46Q, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.414590841 (190h39m30.845670972s) [noise=true , address=104.36.82.3:4799 , piece_id=WDDOZ6UNMGDAF45BAO63GSRVQKABPQNCMLU6XEIHVRVI5HC6VYEQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.437247702 (190h39m30.868327833s) [noise=true , address=104.36.82.34:4565 , piece_id=HF4PBKDU4AWLH3AJ356WG2JWCDQJSQGVH6WLFBCWZFG4PEGJMOUQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.488147614 (190h39m30.919227745s) [noise=true , address=104.36.82.58:4286 , piece_id=PXVVSVTWIA4XS262M4ZWGEXAQFQGMSAWDQDPE42YADZDSSMEKWNQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.489824552 (190h39m30.920904693s) [noise=true , address=77.90.31.13:4647 , piece_id=PLUYPXSAVFOBE4EZMDYE7ELVFBQ32INNTHKT3QDFBZBBHMIIWLSA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.495938310 (190h39m30.927018441s) [noise=true , address=104.36.82.16:4309 , piece_id=S3DTK6LJZGCRFDP55A7W62OHBHQQ3UH75ZSEHJTNPZ3KYUW3QQKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.575349905 (190h39m31.006430036s) [noise=true , address=77.90.31.21:4880 , piece_id=A5JSHMAI7L6BJ4Q63JVINEUUZ5OHZ6K3VYDYPAQHNLMAXSFIGZKA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.710988976 (190h39m31.142069107s) [noise=true , address=104.36.82.51:4058 , piece_id=LJZ5HIM7UYIECRJ4GA7SLAT24LQC5A24LOOA4D7S3IKZWN6IWU7A, seg_pos={0 0}, upload=198338]: Done uploading piece. err: <nil> Aug 16 16:05:36.711016047 (190h39m31.142096178s) [seg_pos={0 0}, upload=198338]: Segment reached optimal threshold of 14 pieces. Aug 16 16:05:36.711318526 (190h39m31.142398657s) [noise=true , address=104.36.82.62:4418 , piece_id=W7AZJR7DHBKQE5Y6EPHAMX7DG4QVC7RN2PVT26AMUTH7QMCWEFGA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnKJpLn4zQr1Z5BTBVSxbHXRknJh41o24v3GKEduhMRuVb1n9): protocol: expected piece hash; context canceled; EOF Aug 16 16:05:36.711387898 (190h39m31.142468029s) [noise=true , address=77.90.31.12:4607 , piece_id=3TULOYF4X54CBCTT55VQH3TB4RAGFTNFU2TOZROLMVNTYWUNV6NQ, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41408->77.90.31.12:4607: use of closed network connection Aug 16 16:05:36.711426964 (190h39m31.142507095s) [noise=true , address=45.91.116.11:4143 , piece_id=G4QEQTMBS662OJCR2C7TWZ7HTCAJ4GTLFLKMHW7ZWEO7HJXENYLA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q): protocol: rpc: dial tcp 45.91.116.11:4143: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:05:36.711444947 (190h39m31.142525078s) [noise=true , address=104.36.82.23:4469 , piece_id=BH326IB35XXJMKSI67NURVLQB2QAIXK4OWWV5IMMIETRZ3Z65QCA, seg_pos={0 0}, upload=198338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv): protocol: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49542->104.36.82.23:4469: use of closed network connection Aug 16 16:05:36.711490996 (190h39m31.142571137s) [seg_pos={0 0}, upload=198338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 16:05:36.711595235 (190h39m31.142675367s) [upload=198338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 16:05:36.711601397 (190h39m31.142681538s) [upload=198338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:05:38.583410084 (190h39m33.014490215s) [upload=198338]: Done uploading segments... Aug 16 16:05:38.583427637 (190h39m33.014507768s) [upload=198338]: Upload finished. err: <nil> Duration: 12.151938168s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.315686274s FAIL (2025-08-16 16:05:33 UTC) orbiter warning: context deadline exceeded Duration: 6.31570983s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [..X.X..X.....X......................X.................X.....] duration: 15.769087458s FAIL (2025-08-16 16:05:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 15.769112916s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.......X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX.......] duration: 5.873161172s FAIL (2025-08-16 16:05:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.87319097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 5.998167497s FAIL (2025-08-16 16:05:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.998197305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.366123492s FAIL (2025-08-16 16:05:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.366156415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [.....X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 27.086596372s FAIL (2025-08-16 16:04:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 27.086619215s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 6.094142297s FAIL (2025-08-16 16:04:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.094174369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.955372603s FAIL (2025-08-16 16:04:45 UTC) orbiter warning: context deadline exceeded Duration: 7.955394965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [........X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX......] duration: 6.130727799s FAIL (2025-08-16 16:04:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.130775751s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.827906771s FAIL (2025-08-16 16:04:29 UTC) orbiter warning: context deadline exceeded Duration: 7.827930396s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 30.545613785s FAIL (2025-08-16 16:04:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 30.545640696s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 6.147993014s FAIL (2025-08-16 16:04:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.148015527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.824258161s FAIL (2025-08-16 16:04:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.824282797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...X.X..X.....X......................X.................X....] duration: 2.013928916s FAIL (2025-08-16 16:04:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 2.013957101s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.389366771s FAIL (2025-08-16 16:04:02 UTC) orbiter warning: context deadline exceeded Duration: 8.38940297s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.........X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX.....] duration: 6.354898703s FAIL (2025-08-16 16:03:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.35492275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 6.056442341s FAIL (2025-08-16 16:03:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.056467991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 16.528617468s FAIL (2025-08-16 16:03:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 16.52864994s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.215144435s FAIL (2025-08-16 16:03:32 UTC) orbiter warning: context deadline exceeded Duration: 6.215166257s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [....X.X..X.....X......................X.................X...] duration: 16.868071758s FAIL (2025-08-16 16:03:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 16.868115562s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 6.133776067s FAIL (2025-08-16 16:03:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.133796024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 5.928088343s FAIL (2025-08-16 16:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.928106488s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.440284773s FAIL (2025-08-16 16:02:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.440306837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..........X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX....] duration: 6.175790343s FAIL (2025-08-16 16:02:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.175831573s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.748314955s FAIL (2025-08-16 16:02:58 UTC) orbiter warning: context deadline exceeded Duration: 6.748340223s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...........X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX...] duration: 10.029108775s FAIL (2025-08-16 16:02:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 10.029126077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 6.649128301s FAIL (2025-08-16 16:02:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.64916899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X..X.....X......................X.................X..] duration: 1.853088877s FAIL (2025-08-16 16:02:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 1.85311659s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 26.73883492s FAIL (2025-08-16 16:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 26.738848997s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.490606969s FAIL (2025-08-16 16:02:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.490639662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.332915946s FAIL (2025-08-16 16:02:04 UTC) orbiter warning: context deadline exceeded Duration: 6.33293868s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [............X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX..] duration: 9.26274898s FAIL (2025-08-16 16:01:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 9.262777704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 19.395097634s FAIL (2025-08-16 16:01:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 19.395124837s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [......X.X..X.....X......................X.................X.] duration: 17.55622518s FAIL (2025-08-16 16:01:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 17.556251149s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.66209587s FAIL (2025-08-16 16:01:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.662514863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [..X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 44.767678121s FAIL (2025-08-16 16:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 44.767694362s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.............X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX.] duration: 22.171181791s FAIL (2025-08-16 16:01:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 22.171210906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 6.09593547s FAIL (2025-08-16 16:01:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.095949897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.648953316s FAIL (2025-08-16 16:01:14 UTC) orbiter warning: context deadline exceeded Duration: 6.648970078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238693457s FAIL (2025-08-16 16:01:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== upload=198245]: Uploading piece... Aug 16 16:00:10.770806034 (190h34m5.201886165s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.770975228 (190h34m5.202055359s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771077693 (190h34m5.202157825s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771274441 (190h34m5.202354572s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771309197 (190h34m5.202389328s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771570006 (190h34m5.202650137s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771594673 (190h34m5.202674804s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771829703 (190h34m5.202909834s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.771875119 (190h34m5.20295527s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772062949 (190h34m5.20314308s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772183270 (190h34m5.203263401s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:10.772563297 (190h34m5.203643428s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Uploading piece... Aug 16 16:00:11.953464880 (190h34m6.384545011s) [upload=198245]: Next returned nil segment Aug 16 16:00:11.953482854 (190h34m6.384562985s) [upload=198245]: Waiting for error group managing segments... Aug 16 16:00:12.941409550 (190h34m7.372489691s) [noise=true , address=79.127.220.101:4076 , piece_id=LJU4XJG4OFORQVRTD7BOOI47GREELV5F5BGHNZ3VUBBP3HS73JYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:13.659968773 (190h34m8.091048904s) [noise=true , address=45.91.116.7:4043 , piece_id=2V7JUZMBVKV6IFF6TSHFUTJLVYC7ZWC3AG2F3RXRYZUGWCVVNO6A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, address:45.91.116.7:4043): protocol: storage node overloaded, request limit: 2000 Aug 16 16:00:14.611558252 (190h34m9.042638383s) [noise=true , address=104.36.82.45:4934 , piece_id=6AMSA3INP3L3ADURWGJDGXTFOFMQ5VSORF3KNGO35ZSPQNYMSP2Q, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:20.670138101 (190h34m15.101218242s) [noise=true , address=66.23.193.175:4557 , piece_id=RDX5DR364JZMUSTWW4HA4P55RNDONJXR676CFUMOG5TO5SJ7JQQA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:22.111492140 (190h34m16.542572311s) [noise=true , address=45.91.116.5:4315 , piece_id=DYNZCXOWKSP6KPULJLUTQHS6JPGPX7WVCESHEH6TTQQIXZM3QZZA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:24.242804738 (190h34m18.673884869s) [noise=true , address=77.90.31.18:4837 , piece_id=5TSI4MM4SHIYCL3PCD3RHYCJR4FXPM7VVVXQNU7ROAZNWHEON4DQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:00:43.969823882 (190h34m38.400904103s) [noise=true , address=77.90.31.14:4695 , piece_id=ZE67SDK22KFKKAONRYDWRRMG266Q24DHKNVLFNE3NTZDI6XILQSQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: <nil> Aug 16 16:01:09.606812307 (190h35m4.037892438s) [noise=true , address=104.36.82.5:4842 , piece_id=DXZQKVGVMH7TGKVZKRFLI7LTEJY7OBQX6GKRAM4T33CZVRJNGX3A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606854288 (190h35m4.037934419s) [noise=true , address=104.36.82.58:4271 , piece_id=M5B7V2YGDPVNZL73XKJH2YKBAGZR7LCKRLCKQGIBMABSRLAHMSWA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:104.36.82.58:4271): protocol: rpc: dial tcp 104.36.82.58:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606884385 (190h35m4.037964516s) [noise=true , address=104.36.82.60:4353 , piece_id=R6OTTIHWGZE2KAS25Z6YCTCWL2RAQKZGTDFI66Y4JYPSK72I7TTA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: rpc: dial tcp 104.36.82.60:4353: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606917428 (190h35m4.037997559s) [noise=true , address=104.36.82.23:4457 , piece_id=DCL3MYX7AVNV2V3KD5DHFDYYLY3AT6OXE3FEHOGIJB4WLCHQ5KOQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: write tcp 188.166.225.170:39772->104.36.82.23:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.606949459 (190h35m4.03802959s) [noise=true , address=104.36.82.36:4629 , piece_id=K3L4U3YD5N6WDP35N2QKPZ3KLNSCR5R4QWEZS4DMQO4YXOBYXCYQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1uxMhTiH446Ku7h1PutqVh7fSCzwKHR7oypYBzGWELLNP7ojf6, address:104.36.82.36:4629): protocol: write tcp 188.166.225.170:59572->104.36.82.36:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607050022 (190h35m4.038130153s) [noise=true , address=104.36.82.3:4799 , piece_id=LIEZASMYJIYFGFSUGAYBQNS5OY6UPGJ7KWCTBWVMLZ5YMRPGEMTQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37514->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607085500 (190h35m4.038165631s) [noise=true , address=77.90.31.19:4867 , piece_id=WFPEU7FO65IMB3JAFZ6P3IPIEEOXAW6QDPGK7VRXFLWTSEQMC52A, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 188.166.225.170:59224->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607125927 (190h35m4.038206059s) [noise=true , address=104.36.82.44:4888 , piece_id=TPTSTGF3IAALD6BRRPENCRWLDCWNBEUFB5T6P64Q733SPGL6YOKA, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: rpc: dial tcp 104.36.82.44:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607138701 (190h35m4.038218832s) [noise=true , address=77.90.31.11:4559 , piece_id=IXHIHQXDLTSVFOPDCAHGYDEI22RBH77SU2DJFZ52MLWZMTANVADQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->77.90.31.11:4559: use of closed network connection Aug 16 16:01:09.607166785 (190h35m4.038246916s) [noise=true , address=104.36.82.22:4543 , piece_id=XTN6FE7SKOCWPE6VQTEIFZFZ6OLPFSJSMJRUNYQUU4DJ7JCZR5UQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: rpc: dial tcp 104.36.82.22:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607185561 (190h35m4.038265692s) [noise=true , address=104.36.82.16:4306 , piece_id=2547BUZR4R5PGGF7MHB2HIWIX42JLKHM4AMNFTSUGYIM55YYYBHQ, seg_pos={0 0}, upload=198245]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:01:09.607211200 (190h35m4.038291332s) [seg_pos={0 0}, upload=198245]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:01:09.843745933 (190h35m4.274826064s) [upload=198245]: Done uploading segments... Aug 16 16:01:09.843772213 (190h35m4.274852344s) [upload=198245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.238707786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 20.43123s FAIL (2025-08-16 16:01:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.431253404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 10.070662844s FAIL (2025-08-16 16:00:57 UTC) orbiter warning: context deadline exceeded Duration: 10.070713981s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X..X.....X......................X.................X] duration: 2.991605592s FAIL (2025-08-16 16:00:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 16:00:46.408121382 (190h34m40.839201513s) [upload=198253]: Starting upload Aug 16 16:00:46.408175154 (190h34m40.839255285s) [upload=198253]: Done starting upload Aug 16 16:00:46.408231292 (190h34m40.839311423s) [upload=198253]: Uploading segments... Aug 16 16:00:46.408295395 (190h34m40.839375526s) [upload=198253]: Got next segment. Inline: true Aug 16 16:00:46.408299562 (190h34m40.839379694s) [upload=198253]: Waiting for error group managing segments... Aug 16 16:00:46.408333848 (190h34m40.839413979s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:00:46.408340020 (190h34m40.839420151s) [upload=198253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:00:46.408343106 (190h34m40.839423237s) [upload=198253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:00:47.863343346 (190h34m42.294423477s) [upload=198253]: Done uploading segments... Aug 16 16:00:47.863363494 (190h34m42.294443615s) [upload=198253]: Upload finished. err: <nil> Duration:* 2.991638404s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.316498887s FAIL (2025-08-16 16:00:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Aug 16 15:59:37.837390867 (190h33m32.268470998s) [upload=198236]: Starting upload Aug 16 15:59:37.837436414 (190h33m32.268516545s) [upload=198236]: Done starting upload Aug 16 15:59:37.837538720 (190h33m32.268618861s) [upload=198236]: Uploading segments... Aug 16 15:59:37.837691732 (190h33m32.268771863s) [upload=198236]: Got next segment. Inline: false Aug 16 15:59:37.837728614 (190h33m32.268808745s) [upload=198236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:37.837732902 (190h33m32.268813033s) [upload=198236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:38.361649988 (190h33m32.792730119s) [seg_pos={0 0}, upload=198236]: Begin upload segment... Aug 16 15:59:38.361691658 (190h33m32.792771789s) [seg_pos={0 0}, upload=198236]: Done begin upload segment. Aug 16 15:59:38.361707678 (190h33m32.792787799s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361715143 (190h33m32.792795274s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.361754588 (190h33m32.792834709s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362100621 (190h33m32.793180812s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362160555 (190h33m32.793240686s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362429620 (190h33m32.793509751s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362466891 (190h33m32.793547022s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.362847159 (190h33m32.79392729s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363127245 (190h33m32.794207366s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363353928 (190h33m32.794434059s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363570172 (190h33m32.794650293s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363790854 (190h33m32.794870985s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.363973394 (190h33m32.795053525s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.364898234 (190h33m32.795978365s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365189130 (190h33m32.796269261s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365483082 (190h33m32.796563213s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.365785942 (190h33m32.796866073s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:38.369941713 (190h33m32.801021854s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Uploading piece... Aug 16 15:59:39.911725534 (190h33m34.342805665s) [upload=198236]: Next returned nil segment Aug 16 15:59:39.911746715 (190h33m34.342826846s) [upload=198236]: Waiting for error group managing segments... Aug 16 15:59:43.200146437 (190h33m37.631226568s) [noise=true , address=79.127.201.215:4346 , piece_id=TCYS4NYP7XSNCENMJCSPJAZEZUCX6QHHZ3TPFRDLKA3PUZQNY6NA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:44.267348096 (190h33m38.698428217s) [noise=true , address=77.90.31.13:4636 , piece_id=HU265NMD4L4UIKZN67WS7PYIICV4P5ZDVTSQVL2W46HMX4M2YI3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.013705251 (190h33m50.444785372s) [noise=true , address=45.91.116.5:4321 , piece_id=HYY25FFGZQTRQ274STZZQ74KXA7ZBABRSUTNANDUGQP75JWCE3BA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 15:59:56.754383567 (190h33m51.185463698s) [noise=true , address=77.90.31.18:4826 , piece_id=HVZ4B75I7OTIK6NU3ZGFELOEQMLL2ND3RSZFUN5D5FXDNXKDFWFQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:03.907277192 (190h33m58.338357334s) [noise=true , address=77.90.31.19:4851 , piece_id=LHPG4VS74QBUZADAHAX733GH24WORSMOKP3Y2T2W4JOT7X7UBEWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:15.919184097 (190h34m10.350264228s) [noise=true , address=77.90.31.15:4705 , piece_id=TB53H75VSWM6OK54C34HLULZV3ILSODYXSVLGOX3UAUPT2WPSRKA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: <nil> Aug 16 16:00:37.191017346 (190h34m31.622097477s) [noise=true , address=104.36.82.23:4463 , piece_id=ZJ57JIXGPWPPH6K64XBABOHDSMNVTBG74SU7WKPZZH5QGXNA3PJA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:50742->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191067272 (190h34m31.622147403s) [noise=true , address=104.36.82.36:4645 , piece_id=S3LZPRROHNV76DRUUPAZVPMWMWED6DL27C66HTCBUB6L6ABPUIQQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12NPZfT1KXFLcfPTjX8zLQ4gpagKYosJbvCTKu9sK7989hDjTPm, address:104.36.82.36:4645): protocol: write tcp 188.166.225.170:44810->104.36.82.36:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191099153 (190h34m31.622179284s) [noise=true , address=104.36.82.59:4317 , piece_id=P6LGK33YYKULVOH4OMLQSXP5GO54YFHVZ3TDIM2GD4V22Q54V6HA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: rpc: dial tcp 104.36.82.59:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191128419 (190h34m31.62220855s) [noise=true , address=104.36.82.3:4795 , piece_id=HWWGAKR3VWRCDEZI2OHW4BMPXSTA2BVVK5S726DYSPKU64T76N2Q, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:49916->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191157055 (190h34m31.622237186s) [noise=true , address=104.36.82.44:4870 , piece_id=N42K3UQMVG3YKYDJADHTSSIP5AZRCO2WXBMOQW45G5XX3C4SU6WQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:33300->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191234522 (190h34m31.622314653s) [noise=true , address=104.36.82.60:4353 , piece_id=NZEVYL7BTUNXICNWGLH7EQXPI3D7BA77EGS222XNTVZS6JWHOYDQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ, address:104.36.82.60:4353): protocol: write tcp 188.166.225.170:39034->104.36.82.60:4353: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191268438 (190h34m31.622348569s) [noise=true , address=77.90.31.11:4567 , piece_id=TDZHEBOFEGYWQBGJMOFC4HITLGUXQVDIPS5NXF7YYWJMXZMMBFPQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 188.166.225.170:45660->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191297373 (190h34m31.622377504s) [noise=true , address=104.36.82.5:4844 , piece_id=Q6UCGCDRI7RP2PATX4WEYB62PIEQECFOI6QF3QSJRH7V24ME3O3A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: rpc: dial tcp 104.36.82.5:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191304346 (190h34m31.622384477s) [noise=true , address=104.36.82.4:4832 , piece_id=MHJNYISLILEBWJD25WOA4VPOHLNT2YEL2C673NPJX4NX7LN6ZIOQ, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 188.166.225.170:38748->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191339794 (190h34m31.622419925s) [noise=true , address=104.36.82.34:4575 , piece_id=5V5JNL45Y7OJQJJOD7VCOJ3H7FBNXL3HFTWAZAAS4R27MDXH2NWA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:12nswehA4jhaChDy8FzzdHuByjK6hPm67sXeLCaQZRXe62SimKh, address:104.36.82.34:4575): protocol: write tcp 188.166.225.170:33494->104.36.82.34:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191343240 (190h34m31.622423381s) [noise=true , address=104.36.82.38:4701 , piece_id=XQFTT2AZ6RY7WLCABGFN2WUFCFZWJXR26SGS35JE2UKC5W52RPEA, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191370302 (190h34m31.622450433s) [noise=true , address=104.36.82.2:4751 , piece_id=QSJ3BKCHORGP5AHOK3LI57EIQS22VUZWRIPMCMYQHMNFRY3NHK7A, seg_pos={0 0}, upload=198236]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: rpc: dial tcp 104.36.82.2:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:37.191397364 (190h34m31.622477495s) [seg_pos={0 0}, upload=198236]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 16:00:37.505911833 (190h34m31.936991974s) [upload=198236]: Done uploading segments... Aug 16 16:00:37.505943022 (190h34m31.937023163s) [upload=198236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.316514898s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..............X......X..XXXXXXXXXX.XX.XXXXXX......X.......XX] duration: 30.40485013s FAIL (2025-08-16 16:00:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== . Aug 16 15:59:57.963918156 (190h33m52.394998287s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964029729 (190h33m52.39510986s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964451335 (190h33m52.395531467s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964727965 (190h33m52.395808086s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.964797899 (190h33m52.39587802s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965042367 (190h33m52.396122498s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965090028 (190h33m52.396170159s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965305831 (190h33m52.396385962s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965343212 (190h33m52.396423344s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965548745 (190h33m52.396628876s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965569796 (190h33m52.396649927s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.965765831 (190h33m52.396845962s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966003606 (190h33m52.397083737s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:57.966271599 (190h33m52.39735173s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Uploading piece... Aug 16 15:59:59.736319114 (190h33m54.167399245s) [upload=198240]: Next returned nil segment Aug 16 15:59:59.736337630 (190h33m54.167417761s) [upload=198240]: Waiting for error group managing segments... Aug 16 16:00:00.028564434 (190h33m54.459644565s) [noise=true , address=104.36.82.6:4874 , piece_id=FI3EZZH5G2O42GFDQGX6ZRLX2KFM55UTNMCJY7AGYS2LPH4WCO3Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.092359868 (190h33m54.523439999s) [noise=true , address=104.36.82.7:4912 , piece_id=QG2PMXGDHI7YDPOWL5NBXX3C2N2EHVY2CL6X3QICJZABHVLJKI3A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.158899656 (190h33m54.589979787s) [noise=true , address=104.36.82.10:4079 , piece_id=B3RYSKZKQPUPL4ZHJ6IAOKGF7XPNJMJPWKEGQ2P3HC2WLCGUJ2XA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:00.274239525 (190h33m54.705319646s) [noise=true , address=104.36.82.15:4258 , piece_id=GJJ3FE23W7G5VYXQ7VFF4F6H4RECYP6I4QTY5UYFMPA2XF6APABA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:08.612621871 (190h34m3.043702012s) [noise=true , address=45.91.116.6:4974 , piece_id=KXQZPRG5V5AJF56XD7YRTHAUOP7RCOACV3TIYDA7R7DACD3LID2A, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:25.194912489 (190h34m19.62599263s) [noise=true , address=45.91.116.9:4263 , piece_id=2TC6SI42AFPDXKYWKKEFWDIJLVLAH2TAQ6SKJYDYGBEEYYTQ3IYQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: <nil> Aug 16 16:00:27.035229138 (190h34m21.466309269s) [noise=true , address=45.91.116.12:4227 , piece_id=H7VBILB3XMAMH3XOVKS57O7J6ZKMAG5LEUB25MHXZPSGUX4ZALUQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035398122 (190h34m21.466478263s) [noise=true , address=45.91.116.4:4939 , piece_id=LK6XVI4CHDVZGHGL4QZIZABS2XCWGRI6DEJDKY7CC5PVEB4V5KAA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 16:00:27.035640335 (190h34m21.466720496s) [noise=true , address=45.91.116.5:4297 , piece_id=AZCIMUM2ATU333E5X224BJEGA4ZDGTXNSWLQ7EWQ2XVRUO3SJXQQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46792->45.91.116.5:4297: use of closed network connection Aug 16 16:00:27.035680382 (190h34m21.466760513s) [noise=true , address=45.91.116.14:4190 , piece_id=JJGVJBLIICUJDJD6R7AQY3DL3SAYONGBIY2NRGLJOFWW6WHFKM6Q, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49854->45.91.116.14:4190: use of closed network connection Aug 16 16:00:27.035716621 (190h34m21.466796752s) [noise=true , address=45.91.116.4:4940 , piece_id=JVCUNP6OX3DFZGTJY6FNCJE55FRR3BT6D3UONINJF4Y6GCTJ3FXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56748->45.91.116.4:4940: use of closed network connection Aug 16 16:00:27.035748352 (190h34m21.466828483s) [noise=true , address=104.36.82.16:4293 , piece_id=DM6MQ7NBGY3QEPF5PKLNOHH2SCKKYBI25TF6PAR4QRX2BPHDVLXQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035901174 (190h34m21.466981305s) [noise=true , address=104.36.82.11:4115 , piece_id=OMI5C3GY4N6YCYVC75KM5ZBYCFPUE3Y4YYSK5PEBZVDIAC4V5QPA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:33266->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:27.035939287 (190h34m21.467019418s) [noise=true , address=45.91.116.7:4011 , piece_id=UXAIDQXZAV7LTIMWELENQWGLBUMLOMQN5WK32XBAGSGPQTIO4MPQ, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43028->45.91.116.7:4011: use of closed network connection Aug 16 16:00:27.035966249 (190h34m21.46704638s) [noise=true , address=77.90.31.14:4663 , piece_id=AQ4UKW62AAVZ3WMTR7GCBNPCLF2U3F2ZUO6RH7VPI27EKSTM4UGA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55318->77.90.31.14:4663: use of closed network connection Aug 16 16:00:27.036250663 (190h34m21.467330794s) [noise=true , address=104.36.82.8:4970 , piece_id=XIPQ7G4S7ETV46ME3HDJTUUXDTNGWS2ZX75VO57NV4R35OGLFZOA, seg_pos={0 0}, upload=198240]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Aug 16 16:00:27.036322741 (190h34m21.467402872s) [seg_pos={0 0}, upload=198240]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 16:00:27.438563272 (190h34m21.869643404s) [upload=198240]: Done uploading segments... Aug 16 16:00:27.438589974 (190h34m21.869670105s) [upload=198240]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.404870811s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 10.095903425s FAIL (2025-08-16 16:00:10 UTC) orbiter warning: context deadline exceeded Duration: 10.095929104s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233237268s FAIL (2025-08-16 16:00:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== NYYG6JIUGSCUXF6GBCFJSX7DA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.249883764 (190h33m4.680963885s) [noise=true , address=77.90.31.14:4685 , piece_id=N5O3YFTSXZ442YPWH3QWQGVG3LDB3SIEO2WWJ73ZI5UOFJWPCOKA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.249934211 (190h33m4.681014342s) [noise=true , address=77.90.31.13:4636 , piece_id=MMK3PD442FNLA4YFHARRIQTGR5FOA5WXG5FEII2ORUFNVEMCOX4A, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.250174129 (190h33m4.68125425s) [noise=true , address=104.36.82.60:4345 , piece_id=OR2YUNB7OVPGJLZKLTGPFWMLGKVUXUNLFBW3KSAMXEYUNI4II4CA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.250240557 (190h33m4.681320688s) [noise=true , address=104.36.82.23:4449 , piece_id=74TBMQCCPPZCKX65YDQJRHN6DB5PKBMSV6SB62QRONG6J5HDKNWA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.250510043 (190h33m4.681590164s) [noise=true , address=104.36.82.58:4293 , piece_id=MJWTB6FCWECCBWNIOPRCYIFOAKZ5BZZ3OF6X5IOCZFQXA75XLOJQ, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.250756434 (190h33m4.681836565s) [noise=true , address=104.36.82.44:4882 , piece_id=3TSLXLJDJUQXGUHMWF45NHTMDMDFQ7HXZLC2ZHSXWGQU3YE6AXYQ, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.250996663 (190h33m4.682076794s) [noise=true , address=79.127.220.101:4076 , piece_id=FGPXAGLCQUHVOYGWNDKEFHUBMC5I7I3X4SPJYUR5RLKI6TRF4XIA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.251344609 (190h33m4.68242474s) [noise=true , address=77.90.31.19:4874 , piece_id=AFLQHLJNDKOHKVKSGNLZSAC23RE3E7YMNX7D3Q6CEZT5YBCONEDQ, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.251715639 (190h33m4.68279577s) [noise=true , address=45.91.116.9:4294 , piece_id=TQR7DANYJVWQLSZIQCLKJ32XMI4ZLRZNRB62ARMB4Y3XFGOADSLA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.252000163 (190h33m4.683080294s) [noise=true , address=45.91.116.6:4990 , piece_id=MOHS4IVOJ2YB5Q6JWDYGF7VYH6LR254RL3UGV675OCRUYAS4NHKA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.252910866 (190h33m4.683990997s) [noise=true , address=45.91.116.11:4131 , piece_id=Z6LECUOZ3BDDLO4IOJHBQWAZQBOQ2TB3GINRALB54ALN25ORLGWQ, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.253204578 (190h33m4.684284709s) [noise=true , address=104.36.82.5:4861 , piece_id=QU7QVB4X6M7MHJ75BI7BPGCG6H2C7VVMNCVUNBIZDP4OE46YMS4A, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.253487048 (190h33m4.684567179s) [noise=true , address=64.49.13.169:4706 , piece_id=P5OTZ5OV4NIVJSROZCMFI77UQQFS27KOG5UGQ7L5IIVSBTQPXQGA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.258057333 (190h33m4.689137464s) [noise=true , address=77.90.31.11:4581 , piece_id=F5XX6HZ7IWHCULM4LBDK4BNKG2MPZ66Z6XW65CBQMAJGT6I2XZDA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:10.258377105 (190h33m4.689457237s) [noise=true , address=45.91.116.4:4950 , piece_id=TFITVTPKHQBIV6NBGPVATGWN7V6SN6QVMHOOGA3COMUSLCPMBENA, seg_pos={0 0}, upload=198231]: Uploading piece... Aug 16 15:59:11.265549667 (190h33m5.696629798s) [noise=true , address=79.127.220.101:4076 , piece_id=FGPXAGLCQUHVOYGWNDKEFHUBMC5I7I3X4SPJYUR5RLKI6TRF4XIA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:16diJ16eDS6LX9gZHdM6zwyMW3UanSXniaBbFUDCVb5H9QSUbZ, address:79.127.220.101:4076): protocol: storage node overloaded, request limit: 2000 Aug 16 15:59:11.402567976 (190h33m5.833648107s) [upload=198231]: Next returned nil segment Aug 16 15:59:11.402586001 (190h33m5.833666122s) [upload=198231]: Waiting for error group managing segments... Aug 16 15:59:16.269352032 (190h33m10.700432153s) [noise=true , address=45.91.116.11:4131 , piece_id=Z6LECUOZ3BDDLO4IOJHBQWAZQBOQ2TB3GINRALB54ALN25ORLGWQ, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:16.760949819 (190h33m11.19202995s) [noise=true , address=77.90.31.13:4636 , piece_id=MMK3PD442FNLA4YFHARRIQTGR5FOA5WXG5FEII2ORUFNVEMCOX4A, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:16.876336519 (190h33m11.30741665s) [noise=true , address=45.91.116.6:4990 , piece_id=MOHS4IVOJ2YB5Q6JWDYGF7VYH6LR254RL3UGV675OCRUYAS4NHKA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:17.051876735 (190h33m11.482956866s) [noise=true , address=45.91.116.9:4294 , piece_id=TQR7DANYJVWQLSZIQCLKJ32XMI4ZLRZNRB62ARMB4Y3XFGOADSLA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:19.336595516 (190h33m13.767675648s) [noise=true , address=77.90.31.14:4685 , piece_id=N5O3YFTSXZ442YPWH3QWQGVG3LDB3SIEO2WWJ73ZI5UOFJWPCOKA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:22.049385125 (190h33m16.480465256s) [noise=true , address=45.91.116.4:4950 , piece_id=TFITVTPKHQBIV6NBGPVATGWN7V6SN6QVMHOOGA3COMUSLCPMBENA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:22.109626821 (190h33m16.540706952s) [noise=true , address=64.49.13.169:4706 , piece_id=P5OTZ5OV4NIVJSROZCMFI77UQQFS27KOG5UGQ7L5IIVSBTQPXQGA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 15:59:24.903915901 (190h33m19.334996032s) [noise=true , address=77.90.31.19:4874 , piece_id=AFLQHLJNDKOHKVKSGNLZSAC23RE3E7YMNX7D3Q6CEZT5YBCONEDQ, seg_pos={0 0}, upload=198231]: Done uploading piece. err: <nil> Aug 16 16:00:09.143885023 (190h34m3.574965154s) [noise=true , address=104.36.82.39:4746 , piece_id=6D4P4MFUTVEXBGYD5VA2LHES3CNNYYG6JIUGSCUXF6GBCFJSX7DA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:12SBP5cx63KB8oMuzhuq78a1V88januH2X5K3MzJGdXxFvXD4VM, address:104.36.82.39:4746): protocol: write tcp 188.166.225.170:36032->104.36.82.39:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.143945779 (190h34m3.57502591s) [noise=true , address=104.36.82.60:4345 , piece_id=OR2YUNB7OVPGJLZKLTGPFWMLGKVUXUNLFBW3KSAMXEYUNI4II4CA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: rpc: dial tcp 104.36.82.60:4345: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.143980636 (190h34m3.575060767s) [noise=true , address=104.36.82.38:4702 , piece_id=JENYZZBD4MUZGHWXHUDB53L66XTPLMUHGLL3TOXMHSXIVTDOGJDQ, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: rpc: dial tcp 104.36.82.38:4702: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144043697 (190h34m3.575123828s) [noise=true , address=104.36.82.59:4321 , piece_id=ALHOMAHOJL3Y7URYRKPCLMFNPLLEPSBY26XTXFOH2MIFKXZHWKZQ, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc, address:104.36.82.59:4321): protocol: rpc: dial tcp 104.36.82.59:4321: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144109322 (190h34m3.575189453s) [noise=true , address=104.36.82.23:4449 , piece_id=74TBMQCCPPZCKX65YDQJRHN6DB5PKBMSV6SB62QRONG6J5HDKNWA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:104.36.82.23:4449): protocol: rpc: dial tcp 104.36.82.23:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144146133 (190h34m3.575226264s) [noise=true , address=104.36.82.44:4882 , piece_id=3TSLXLJDJUQXGUHMWF45NHTMDMDFQ7HXZLC2ZHSXWGQU3YE6AXYQ, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:1TU7roqQsKNGvNeqmWdVxMregk9Dqn3YZpfdvK6xp7c3zYvccc, address:104.36.82.44:4882): protocol: rpc: dial tcp 104.36.82.44:4882: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144210335 (190h34m3.575290467s) [noise=true , address=104.36.82.58:4293 , piece_id=MJWTB6FCWECCBWNIOPRCYIFOAKZ5BZZ3OF6X5IOCZFQXA75XLOJQ, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: rpc: dial tcp 104.36.82.58:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144511592 (190h34m3.575591723s) [noise=true , address=104.36.82.5:4861 , piece_id=QU7QVB4X6M7MHJ75BI7BPGCG6H2C7VVMNCVUNBIZDP4OE46YMS4A, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:55298->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144596975 (190h34m3.575677106s) [noise=true , address=77.90.31.11:4581 , piece_id=F5XX6HZ7IWHCULM4LBDK4BNKG2MPZ66Z6XW65CBQMAJGT6I2XZDA, seg_pos={0 0}, upload=198231]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: read tcp 188.166.225.170:59508->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 16:00:09.144647873 (190h34m3.575728004s) [seg_pos={0 0}, upload=198231]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 16:00:09.375284129 (190h34m3.80636426s) [upload=198231]: Done uploading segments... Aug 16 16:00:09.375317724 (190h34m3.806397855s) [upload=198231]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.23327517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 16.798361326s FAIL (2025-08-16 16:00:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.798388216s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)