Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................................X] duration: 1.39121363s FAIL (2025-07-27 00:23:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.391241093s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX] duration: 30.233287166s FAIL (2025-07-27 00:23:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== FAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:23:26.878833834 (60h47m36.601396864s) [upload=58298]: Starting upload Jul 27 00:23:26.878889149 (60h47m36.60145219s) [upload=58298]: Done starting upload Jul 27 00:23:26.878958201 (60h47m36.601521241s) [upload=58298]: Uploading segments... Jul 27 00:23:26.879205003 (60h47m36.601768043s) [upload=58298]: Got next segment. Inline: false Jul 27 00:23:26.879232696 (60h47m36.601795736s) [upload=58298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:23:26.879237807 (60h47m36.601800887s) [upload=58298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:23:27.117701870 (60h47m36.84026491s) [seg_pos={0 0}, upload=58298]: Begin upload segment... Jul 27 00:23:27.117772716 (60h47m36.840335756s) [seg_pos={0 0}, upload=58298]: Done begin upload segment. Jul 27 00:23:27.117840324 (60h47m36.840403364s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.117860343 (60h47m36.840423383s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118423991 (60h47m36.840987031s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118698997 (60h47m36.841262037s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119055539 (60h47m36.841618579s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119302452 (60h47m36.841865482s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119503998 (60h47m36.842067028s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119695213 (60h47m36.842258253s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119894365 (60h47m36.842457405s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120153350 (60h47m36.84271639s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120380224 (60h47m36.842943254s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120589725 (60h47m36.843152765s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120830175 (60h47m36.843393215s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121036910 (60h47m36.84359995s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121246862 (60h47m36.843809892s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121523792 (60h47m36.844086822s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:28.854018534 (60h47m38.576581564s) [upload=58298]: Next returned nil segment Jul 27 00:23:28.854052358 (60h47m38.576615398s) [upload=58298]: Waiting for error group managing segments... Jul 27 00:23:29.129523117 (60h47m38.852086157s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155007436 (60h47m38.877570476s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155488357 (60h47m38.878051397s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.157090652 (60h47m38.879653692s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.222674814 (60h47m38.945237854s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.223964532 (60h47m38.946527572s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.232334866 (60h47m38.954897907s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.236272311 (60h47m38.958835361s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.302653628 (60h47m39.025216708s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.356275302 (60h47m39.078838332s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:31.238097536 (60h47m40.960660576s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:56.226454714 (60h48m5.949017754s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:51898->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226501363 (60h48m5.949064403s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226622195 (60h48m5.949185235s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39816->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226649257 (60h48m5.949212297s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226667200 (60h48m5.94923024s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226683812 (60h48m5.949246852s) [seg_pos={0 0}, upload=58298]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:23:56.458682075 (60h48m6.181245115s) [upload=58298]: Done uploading segments... Jul 27 00:23:56.458702984 (60h48m6.181266025s) [upload=58298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.233300991s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........] duration: 8.186698978s FAIL (2025-07-27 00:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.186719747s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.........] duration: 28.790575933s FAIL (2025-07-27 00:23:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.790597585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.245761056s FAIL (2025-07-27 00:23:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== QOVEGNAHH7YHZ6JLWA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.358889046 (60h46m46.081452086s) [noise=true , address=45.91.116.14:4206 , piece_id=UEG65X4ZHLXE5A4MVVO2NB32IIWFU6TYEZ6UM2HAN3LSZQ6P5AWQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.359306165 (60h46m46.081869215s) [noise=true , address=77.90.31.16:4736 , piece_id=OSQHUQ75WTKSKEVA4UW4PDPGYFB55562OQB6KHP7OE4GVKMKMR4A, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.359532648 (60h46m46.082095688s) [seg_pos={0 0}, upload=58286]: Done begin upload segment. Jul 27 00:22:36.359558287 (60h46m46.082121327s) [noise=true , address=79.127.205.233:4645 , piece_id=TA5YMHFL4P2PDSZJUMBENDS7JRUO5YA4XSWTHHF4MXZE4HYQSKIQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.360857503 (60h46m46.083420543s) [noise=true , address=104.36.82.52:4103 , piece_id=FESHL3LIBNB4ZN522RGNKAG65K5LTUCQ5G3DCWR5ESHBOSE4NA4Q, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361150514 (60h46m46.083713554s) [noise=true , address=77.90.31.10:4535 , piece_id=2RLHLSENNQBPKCLPDZEOAXXL4HAVZEDKXOKC3XEYJBYG4OMZ2O2Q, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361476588 (60h46m46.084039628s) [noise=true , address=77.90.31.11:4584 , piece_id=X6MXKANUP76HLLF2DVWB27TNQ5AFE2YBK7TMZE4ZGMKY5HGOMWJA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361767975 (60h46m46.084331015s) [noise=true , address=104.36.82.56:4234 , piece_id=HPU55E7EGQPQ2MO6EOVEQRTZ5PHB2EAC353CRPIBB3JD7JKAGKEA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361974490 (60h46m46.08453753s) [noise=true , address=79.127.205.245:4218 , piece_id=JVBNWJFETGQACITDTMN4AAPTK4DWKDHGKMRQYX3U2AXBDKZ3PIXQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362101503 (60h46m46.084664543s) [noise=true , address=79.127.201.215:4350 , piece_id=CHEYZPZKM7VQCBOBENT6BSWJHYJPYUIYUF54NGBJ4L2EUEG3EACQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362583535 (60h46m46.085146575s) [noise=true , address=79.127.219.38:4001 , piece_id=UE7IV4A3PL7MMWVIFWTE43AFC4EJT62QYZSLYODJQVBXNG3IXEQQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362912635 (60h46m46.085475675s) [noise=true , address=104.36.82.47:4542 , piece_id=THLRASBMMBVBKR2DPLDFGS4V56U3K4PG6VBUSZDPRBRXSMO6FOTQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362921011 (60h46m46.085484051s) [noise=true , address=79.127.205.235:4677 , piece_id=5RAX26ESH7U7BADWHVD254Y75CNKICNLZZCATNEEUUD3AWVXW2MQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362956859 (60h46m46.085519899s) [noise=true , address=79.127.220.97:4361 , piece_id=NNVVZFQAVDSGTOXMMDN5CFX3432I26GZAR27BUYTO4RXQYSXFKYQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363233869 (60h46m46.085796909s) [noise=true , address=77.90.31.18:4815 , piece_id=7MHGJR5HB2YKHPBKYXLV2SWE3FIORSX55DLQWBJZIC5Q3EE7TMVA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363246133 (60h46m46.085809173s) [noise=true , address=104.36.82.55:4225 , piece_id=EDEFOFIJEPIR77TKHGI6TFFDZD76577E2RBABYONANAE2ILK6YVQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363537079 (60h46m46.086100119s) [noise=true , address=64.49.13.140:4508 , piece_id=SHX2AKPUOHLEKH2OY3TKMPQUIQEIAXFFMX4LZVGZT5TJ2BN7VK3A, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363868323 (60h46m46.086431363s) [noise=true , address=79.127.220.101:4041 , piece_id=JIEZ6K5U2BRK2MIY6LFJH6OWWHX4ACZ4HIUQLFVLY7GOD67RXBNQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:37.447665282 (60h46m47.170228332s) [upload=58286]: Next returned nil segment Jul 27 00:22:37.447692194 (60h46m47.170255234s) [upload=58286]: Waiting for error group managing segments... Jul 27 00:22:37.501411946 (60h46m47.223974986s) [noise=true , address=79.127.220.97:4361 , piece_id=NNVVZFQAVDSGTOXMMDN5CFX3432I26GZAR27BUYTO4RXQYSXFKYQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:22:37.745347919 (60h46m47.467910959s) [noise=true , address=79.127.205.234:4606 , piece_id=SYDVXXEWEHM25RAQEGWH47HFZZ7ZOHXDMNQOVEGNAHH7YHZ6JLWA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:37.747484648 (60h46m47.470047688s) [noise=true , address=79.127.205.245:4218 , piece_id=JVBNWJFETGQACITDTMN4AAPTK4DWKDHGKMRQYX3U2AXBDKZ3PIXQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:37.781937336 (60h46m47.504500376s) [noise=true , address=79.127.205.233:4645 , piece_id=TA5YMHFL4P2PDSZJUMBENDS7JRUO5YA4XSWTHHF4MXZE4HYQSKIQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.030972264 (60h46m47.753535304s) [noise=true , address=79.127.205.235:4677 , piece_id=5RAX26ESH7U7BADWHVD254Y75CNKICNLZZCATNEEUUD3AWVXW2MQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.250917325 (60h46m47.973480365s) [noise=true , address=79.127.219.38:4001 , piece_id=UE7IV4A3PL7MMWVIFWTE43AFC4EJT62QYZSLYODJQVBXNG3IXEQQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.332626600 (60h46m48.05518964s) [noise=true , address=77.90.31.18:4815 , piece_id=7MHGJR5HB2YKHPBKYXLV2SWE3FIORSX55DLQWBJZIC5Q3EE7TMVA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:40.304136871 (60h46m50.026699911s) [noise=true , address=79.127.220.101:4041 , piece_id=JIEZ6K5U2BRK2MIY6LFJH6OWWHX4ACZ4HIUQLFVLY7GOD67RXBNQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:41.704307309 (60h46m51.426870349s) [noise=true , address=64.49.13.140:4508 , piece_id=SHX2AKPUOHLEKH2OY3TKMPQUIQEIAXFFMX4LZVGZT5TJ2BN7VK3A, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:49.259512908 (60h46m58.982075948s) [noise=true , address=45.91.116.14:4206 , piece_id=UEG65X4ZHLXE5A4MVVO2NB32IIWFU6TYEZ6UM2HAN3LSZQ6P5AWQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:52.112812417 (60h47m1.835375457s) [noise=true , address=79.127.201.215:4350 , piece_id=CHEYZPZKM7VQCBOBENT6BSWJHYJPYUIYUF54NGBJ4L2EUEG3EACQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:23:35.254588588 (60h47m44.977151628s) [noise=true , address=77.90.31.16:4736 , piece_id=OSQHUQ75WTKSKEVA4UW4PDPGYFB55562OQB6KHP7OE4GVKMKMR4A, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254765656 (60h47m44.977328686s) [noise=true , address=104.36.82.52:4103 , piece_id=FESHL3LIBNB4ZN522RGNKAG65K5LTUCQ5G3DCWR5ESHBOSE4NA4Q, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254823567 (60h47m44.977386607s) [noise=true , address=104.36.82.47:4542 , piece_id=THLRASBMMBVBKR2DPLDFGS4V56U3K4PG6VBUSZDPRBRXSMO6FOTQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254862192 (60h47m44.977425222s) [noise=true , address=104.36.82.55:4225 , piece_id=EDEFOFIJEPIR77TKHGI6TFFDZD76577E2RBABYONANAE2ILK6YVQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 188.166.225.170:53120->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254922136 (60h47m44.977485166s) [noise=true , address=104.36.82.56:4234 , piece_id=HPU55E7EGQPQ2MO6EOVEQRTZ5PHB2EAC353CRPIBB3JD7JKAGKEA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254977973 (60h47m44.977541013s) [noise=true , address=77.90.31.10:4535 , piece_id=2RLHLSENNQBPKCLPDZEOAXXL4HAVZEDKXOKC3XEYJBYG4OMZ2O2Q, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: rpc: dial tcp 77.90.31.10:4535: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:23:35.255018531 (60h47m44.977581571s) [noise=true , address=77.90.31.11:4584 , piece_id=X6MXKANUP76HLLF2DVWB27TNQ5AFE2YBK7TMZE4ZGMKY5HGOMWJA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: write tcp 188.166.225.170:51642->77.90.31.11:4584: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:23:35.255039040 (60h47m44.97760207s) [seg_pos={0 0}, upload=58286]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:23:35.498418277 (60h47m45.220981307s) [upload=58286]: Done uploading segments... Jul 27 00:23:35.498445649 (60h47m45.221008689s) [upload=58286]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.245775625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.....X......X...XX..XX.X.XX...X...............XX......X...X] duration: 1m0.001072982s FAIL (2025-07-27 00:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001100766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXX] duration: 30.233196508s FAIL (2025-07-27 00:23:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.233223029s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXX] duration: 30.234025965s FAIL (2025-07-27 00:23:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== n closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301966701 (60h47m5.024529751s) [seg_pos={0 0}, upload=58285]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:55.535080726 (60h47m5.257643776s) [upload=58285]: Done uploading segments... Jul 27 00:22:55.535111805 (60h47m5.257674845s) [upload=58285]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:22:56.419177168 (60h47m6.141740228s) [upload=58291]: Starting upload Jul 27 00:22:56.419245739 (60h47m6.141808779s) [upload=58291]: Done starting upload Jul 27 00:22:56.419303629 (60h47m6.14186667s) [upload=58291]: Uploading segments... Jul 27 00:22:56.419440371 (60h47m6.142003411s) [upload=58291]: Got next segment. Inline: false Jul 27 00:22:56.419485077 (60h47m6.142048117s) [upload=58291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:56.419489145 (60h47m6.142052185s) [upload=58291]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:56.658833663 (60h47m6.381396703s) [seg_pos={0 0}, upload=58291]: Begin upload segment... Jul 27 00:22:56.658891664 (60h47m6.381454704s) [seg_pos={0 0}, upload=58291]: Done begin upload segment. Jul 27 00:22:56.658915349 (60h47m6.381478389s) [noise=true , address=77.90.31.16:4734 , piece_id=F7UPDL7MWZ4WNGMNIINOLBQKYQUYXE3MOXZ56NL3YBCI4CCFEXJA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.658926670 (60h47m6.381489721s) [noise=true , address=45.91.116.10:4083 , piece_id=ASGAJYDCPW5FG5KHPIC2P7XBXTR23ADNPKINBCEMF6UH4L3JCCSQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.658967740 (60h47m6.38153078s) [noise=true , address=45.91.116.12:4227 , piece_id=RBOHG2EVNANNCO4JKZDF54I5TLP2Z6VUE7CN2N7XZKD53P4KWV7A, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659311056 (60h47m6.381874086s) [noise=true , address=77.90.31.11:4555 , piece_id=DU7BRWRRNAFGVAKVEV452BUBAQBDSMTNNH3BDJDDXFAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659389747 (60h47m6.381952787s) [noise=true , address=104.36.82.3:4767 , piece_id=5C7TXKX3ZEOESJH5JHAT6FHYXLQR3MJ7A2GXG65N7ANUZFDBIMJA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659789111 (60h47m6.382352151s) [noise=true , address=104.36.82.16:4294 , piece_id=JFUDEUHLS5HSWWHHNHKYGQPL5PGXMWQSDY5CJ33HTITJW3FW7CSA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660189837 (60h47m6.382752877s) [noise=true , address=77.90.31.12:4589 , piece_id=KOGBKJ3T33RKFOZE64VNOGOFVQSQTCEY25P7VSCT53VOVS2QKIYQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660536962 (60h47m6.383100012s) [noise=true , address=104.36.82.2:4730 , piece_id=QJFX6ATBI43Q5PISUZ3DYANVZLGJUJMXTQX5FHAOXOAJ3G4DEOMA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660932729 (60h47m6.383495769s) [noise=true , address=45.91.116.14:4190 , piece_id=7SWEXNFQQV6P5TGO45ZRWIRBIKAGXXH2DXWI4YM4PSQ5QHSUZOHA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661252481 (60h47m6.383815521s) [noise=true , address=104.36.82.15:4258 , piece_id=SIPLDNRTFDKWXVE5EP2JTVEDSY56RKALOUESGRZTKKIXK44CBEQA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661491148 (60h47m6.384054188s) [noise=true , address=45.91.116.8:4045 , piece_id=OA5D77Z77FC5MFIZHS72ELP7ULW5YFL2GHBRRFK744KVFCAZCD4Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661820648 (60h47m6.384383688s) [noise=true , address=77.90.31.15:4699 , piece_id=CVCTE2PSSO2G4AEOQ3QPZ6KEDXDVQGKVBWXJLTSXCFETF2YSPDEQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662040749 (60h47m6.384603789s) [noise=true , address=45.91.116.8:4047 , piece_id=7JUJ2TWF5D2PIFI2HLK32EZ3S7PXC44P27FX3ZXUC7XQOWUN67QA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662292952 (60h47m6.384855992s) [noise=true , address=45.91.116.7:4009 , piece_id=3LZBGS5KMBZGKF2QVU763K5JCLNY5TFHPJ3WH2AC47OO4NQCUNHQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662324492 (60h47m6.384887542s) [noise=true , address=45.91.116.6:4975 , piece_id=A5EARY6IPO5EK3ELDKXFX6WXPLEJ4LWQGWHXPGGX65K7O2IDQ32Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662640596 (60h47m6.385203636s) [noise=true , address=45.91.116.5:4298 , piece_id=WCSH3WKWQPHN56HCCABOOWZ7664QM6RZYTGUNW362OEPQLFBSNYQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:58.385865324 (60h47m8.108428364s) [upload=58291]: Next returned nil segment Jul 27 00:22:58.385883758 (60h47m8.108446798s) [upload=58291]: Waiting for error group managing segments... Jul 27 00:22:58.662217739 (60h47m8.384780779s) [noise=true , address=45.91.116.12:4227 , piece_id=RBOHG2EVNANNCO4JKZDF54I5TLP2Z6VUE7CN2N7XZKD53P4KWV7A, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.682691902 (60h47m8.405254942s) [noise=true , address=45.91.116.7:4009 , piece_id=3LZBGS5KMBZGKF2QVU763K5JCLNY5TFHPJ3WH2AC47OO4NQCUNHQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.683459410 (60h47m8.40602244s) [noise=true , address=45.91.116.8:4045 , piece_id=OA5D77Z77FC5MFIZHS72ELP7ULW5YFL2GHBRRFK744KVFCAZCD4Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.692085685 (60h47m8.414648725s) [noise=true , address=45.91.116.8:4047 , piece_id=7JUJ2TWF5D2PIFI2HLK32EZ3S7PXC44P27FX3ZXUC7XQOWUN67QA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.758650774 (60h47m8.481213814s) [noise=true , address=45.91.116.6:4975 , piece_id=A5EARY6IPO5EK3ELDKXFX6WXPLEJ4LWQGWHXPGGX65K7O2IDQ32Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.762657199 (60h47m8.485220239s) [noise=true , address=45.91.116.10:4083 , piece_id=ASGAJYDCPW5FG5KHPIC2P7XBXTR23ADNPKINBCEMF6UH4L3JCCSQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.829026684 (60h47m8.551589724s) [noise=true , address=45.91.116.14:4190 , piece_id=7SWEXNFQQV6P5TGO45ZRWIRBIKAGXXH2DXWI4YM4PSQ5QHSUZOHA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.855458056 (60h47m8.578021086s) [noise=true , address=45.91.116.5:4298 , piece_id=WCSH3WKWQPHN56HCCABOOWZ7664QM6RZYTGUNW362OEPQLFBSNYQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.878029681 (60h47m8.600592721s) [noise=true , address=77.90.31.15:4699 , piece_id=CVCTE2PSSO2G4AEOQ3QPZ6KEDXDVQGKVBWXJLTSXCFETF2YSPDEQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:23:00.182861607 (60h47m9.905424647s) [noise=true , address=77.90.31.12:4589 , piece_id=KOGBKJ3T33RKFOZE64VNOGOFVQSQTCEY25P7VSCT53VOVS2QKIYQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:23:25.766371949 (60h47m35.488934989s) [noise=true , address=104.36.82.2:4730 , piece_id=QJFX6ATBI43Q5PISUZ3DYANVZLGJUJMXTQX5FHAOXOAJ3G4DEOMA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:36976->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766427906 (60h47m35.488990936s) [noise=true , address=104.36.82.15:4258 , piece_id=SIPLDNRTFDKWXVE5EP2JTVEDSY56RKALOUESGRZTKKIXK44CBEQA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766545671 (60h47m35.489108711s) [noise=true , address=104.36.82.3:4767 , piece_id=5C7TXKX3ZEOESJH5JHAT6FHYXLQR3MJ7A2GXG65N7ANUZFDBIMJA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766580197 (60h47m35.489143237s) [noise=true , address=77.90.31.16:4734 , piece_id=F7UPDL7MWZ4WNGMNIINOLBQKYQUYXE3MOXZ56NL3YBCI4CCFEXJA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:52150->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766639831 (60h47m35.489202871s) [noise=true , address=104.36.82.16:4294 , piece_id=JFUDEUHLS5HSWWHHNHKYGQPL5PGXMWQSDY5CJ33HTITJW3FW7CSA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 188.166.225.170:47750->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766687352 (60h47m35.489250432s) [noise=true , address=77.90.31.11:4555 , piece_id=DU7BRWRRNAFGVAKVEV452BUBAQBDSMTNNH3BDJDDXFAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.234042968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.833626699s FAIL (2025-07-27 00:23:13 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.833656737s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXX] duration: 30.235907451s FAIL (2025-07-27 00:22:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 50860->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994355370 (60h46m15.71691841s) [noise=true , address=104.36.82.12:4150 , piece_id=KNB4EILZ4Z36S7REXMYE3V5BFRJN3LTSO5SSM2MQSQXKJK5RHZEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:22:05.994375438 (60h46m15.716938479s) [seg_pos={0 0}, upload=58272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:22:06.225001157 (60h46m15.947564197s) [upload=58272]: Done uploading segments... Jul 27 00:22:06.225029662 (60h46m15.947592702s) [upload=58272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:22:25.944858424 (60h46m35.667421454s) [upload=58285]: Starting upload Jul 27 00:22:25.944914481 (60h46m35.667477521s) [upload=58285]: Done starting upload Jul 27 00:22:25.945028309 (60h46m35.667591349s) [upload=58285]: Uploading segments... Jul 27 00:22:25.945194817 (60h46m35.667757857s) [upload=58285]: Got next segment. Inline: false Jul 27 00:22:25.945239253 (60h46m35.667802283s) [upload=58285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:25.945244342 (60h46m35.667807382s) [upload=58285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:26.178196499 (60h46m35.900759539s) [seg_pos={0 0}, upload=58285]: Begin upload segment... Jul 27 00:22:26.178252647 (60h46m35.900815687s) [seg_pos={0 0}, upload=58285]: Done begin upload segment. Jul 27 00:22:26.178273276 (60h46m35.900836317s) [noise=true , address=104.36.82.14:4221 , piece_id=TOSIDO6LOV6ZKD4GJHHQ5ZJVDT6NO3QYF7R5QUDIP3ROL7L5BYGA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.178620932 (60h46m35.901183962s) [noise=true , address=45.91.116.8:4045 , piece_id=XCCFCCDKOQAEEPZOF2DYJJFX6LEIGA5W6GSOHPNROPFLWFZNURWQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.178878284 (60h46m35.901441324s) [noise=true , address=77.90.31.15:4699 , piece_id=JZAQWBZ2KYCJRPM3ATBARY5BYFA73EZYTTE4CA5D75VC7MA5IZNQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179154773 (60h46m35.901717813s) [noise=true , address=45.91.116.12:4225 , piece_id=MYW4BZYGMCI6UJUKD7GWIOK6DXVEKBYVY5KBE5XEJX2DH6YQCM2Q, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179431232 (60h46m35.901994262s) [noise=true , address=77.90.31.13:4626 , piece_id=RI7XNQ4QTZMKY5R5OOIC5EZ2Q4S5UJGSEMJGRMRYFSAM5CSXXIRQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179672303 (60h46m35.902235333s) [noise=true , address=45.91.116.13:4153 , piece_id=TYQ3XN3MKX3MCCWKZ7EDZL63EXU4FLZF76C2AUBFQPNBPPXDAOKQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.179916270 (60h46m35.90247931s) [noise=true , address=77.90.31.17:4771 , piece_id=XBGFPEY2UH76YR7X24BAAYOTL7FKQCIZZPRP5QXHI3ZLLZOZDUAQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180229179 (60h46m35.902792219s) [noise=true , address=104.36.82.5:4839 , piece_id=DSKFFCUDEKT6OLFW5C7OIAUCOZLDNADRMUAQG2ESI4NMFFZJYCSQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180454760 (60h46m35.90301779s) [noise=true , address=104.36.82.9:4006 , piece_id=SQXU2TH66A57KSAM3DOZUXLK642ALKFSPCGD4U223RB3SRTDNDCA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180660795 (60h46m35.903223835s) [noise=true , address=45.91.116.5:4297 , piece_id=LMPWPXW7ITIIHRRSHULGNACRQ5PKLMPIYVBP4GOEI5YDKEVPKQIQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.180890554 (60h46m35.903453594s) [noise=true , address=77.90.31.12:4590 , piece_id=J4B75TSVQ4Q5J2ULRT7XGJDFFXJMSCSQLGLTMQFKWJIWMCRIWWBQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.181107900 (60h46m35.90367094s) [noise=true , address=45.91.116.6:4974 , piece_id=YRFYE64PQ43RUINDBVERBOPNUADHPWCVWZZAG7LO6M36KVHMJS4Q, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.181360132 (60h46m35.903923172s) [noise=true , address=45.91.116.4:4937 , piece_id=2AXYYLFCUIZA4AZCK2N4OGF36XH4RJWOU6LI2M244MBJDS5YOWNA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.181569273 (60h46m35.904132313s) [noise=true , address=77.90.31.19:4842 , piece_id=D2FQTUUCLPBBPCXANQEJE4NORTUXK7MSRTEFKHQZNVAILCBXHEFQ, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.182070782 (60h46m35.904633812s) [noise=true , address=45.91.116.10:4081 , piece_id=A2TMO3SWLLVJWIIUPR7V5AW63JBAQYV4KEZNPTSF6CBVH64DI54A, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:26.182540371 (60h46m35.905103411s) [noise=true , address=45.91.116.7:4009 , piece_id=644K4VRNKEAR2PP5JBKNQSNFLVDBYH7KWPNGOPZRMNVIRDLMTEVA, seg_pos={0 0}, upload=58285]: Uploading piece... Jul 27 00:22:27.910590149 (60h46m37.633153189s) [upload=58285]: Next returned nil segment Jul 27 00:22:27.910609567 (60h46m37.633172607s) [upload=58285]: Waiting for error group managing segments... Jul 27 00:22:28.199559705 (60h46m37.922122745s) [noise=true , address=45.91.116.6:4974 , piece_id=YRFYE64PQ43RUINDBVERBOPNUADHPWCVWZZAG7LO6M36KVHMJS4Q, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.208008390 (60h46m37.93057142s) [noise=true , address=45.91.116.4:4937 , piece_id=2AXYYLFCUIZA4AZCK2N4OGF36XH4RJWOU6LI2M244MBJDS5YOWNA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.215670731 (60h46m37.938233771s) [noise=true , address=45.91.116.13:4153 , piece_id=TYQ3XN3MKX3MCCWKZ7EDZL63EXU4FLZF76C2AUBFQPNBPPXDAOKQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.259150630 (60h46m37.98171367s) [noise=true , address=45.91.116.5:4297 , piece_id=LMPWPXW7ITIIHRRSHULGNACRQ5PKLMPIYVBP4GOEI5YDKEVPKQIQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.283540055 (60h46m38.006103095s) [noise=true , address=45.91.116.8:4045 , piece_id=XCCFCCDKOQAEEPZOF2DYJJFX6LEIGA5W6GSOHPNROPFLWFZNURWQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.301075536 (60h46m38.023638576s) [noise=true , address=77.90.31.15:4699 , piece_id=JZAQWBZ2KYCJRPM3ATBARY5BYFA73EZYTTE4CA5D75VC7MA5IZNQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.303189240 (60h46m38.02575228s) [noise=true , address=45.91.116.7:4009 , piece_id=644K4VRNKEAR2PP5JBKNQSNFLVDBYH7KWPNGOPZRMNVIRDLMTEVA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.342583963 (60h46m38.065147003s) [noise=true , address=45.91.116.12:4225 , piece_id=MYW4BZYGMCI6UJUKD7GWIOK6DXVEKBYVY5KBE5XEJX2DH6YQCM2Q, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:28.371911185 (60h46m38.094474225s) [noise=true , address=45.91.116.10:4081 , piece_id=A2TMO3SWLLVJWIIUPR7V5AW63JBAQYV4KEZNPTSF6CBVH64DI54A, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:29.697862379 (60h46m39.42042542s) [noise=true , address=77.90.31.13:4626 , piece_id=RI7XNQ4QTZMKY5R5OOIC5EZ2Q4S5UJGSEMJGRMRYFSAM5CSXXIRQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:29.749540456 (60h46m39.472103496s) [noise=true , address=77.90.31.19:4842 , piece_id=D2FQTUUCLPBBPCXANQEJE4NORTUXK7MSRTEFKHQZNVAILCBXHEFQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:29.891724216 (60h46m39.614287256s) [noise=true , address=77.90.31.12:4590 , piece_id=J4B75TSVQ4Q5J2ULRT7XGJDFFXJMSCSQLGLTMQFKWJIWMCRIWWBQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: <nil> Jul 27 00:22:55.301704308 (60h47m5.024267348s) [noise=true , address=104.36.82.5:4839 , piece_id=DSKFFCUDEKT6OLFW5C7OIAUCOZLDNADRMUAQG2ESI4NMFFZJYCSQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:44334->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301755897 (60h47m5.024318937s) [noise=true , address=77.90.31.17:4771 , piece_id=XBGFPEY2UH76YR7X24BAAYOTL7FKQCIZZPRP5QXHI3ZLLZOZDUAQ, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301790393 (60h47m5.024353433s) [noise=true , address=104.36.82.9:4006 , piece_id=SQXU2TH66A57KSAM3DOZUXLK642ALKFSPCGD4U223RB3SRTDNDCA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:34390->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301934369 (60h47m5.024497419s) [noise=true , address=104.36.82.14:4221 , piece_id=TOSIDO6LOV6ZKD4GJHHQ5ZJVDT6NO3QYF7R5QUDIP3ROL7L5BYGA, seg_pos={0 0}, upload=58285]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301966701 (60h47m5.024529751s) [seg_pos={0 0}, upload=58285]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:55.535080726 (60h47m5.257643776s) [upload=58285]: Done uploading segments... Jul 27 00:22:55.535111805 (60h47m5.257674845s) [upload=58285]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235925155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXX] duration: 30.235081954s FAIL (2025-07-27 00:22:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235114888s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230520689s FAIL (2025-07-27 00:22:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== (60h45m45.622374518s) [noise=true , address=79.127.201.215:4347 , piece_id=RVPEXDTEBGLYUOFLW4SUWV3H3BRPWPCHSS5AOH6OHJFGHKPZLCUA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.899861905 (60h45m45.622424935s) [noise=true , address=104.36.82.53:4123 , piece_id=5T64IZEVUHWPT3RU2XWC7C75FJPMJSXHGXIQV6HGWK3RGJA7DGJA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900123364 (60h45m45.622686394s) [noise=true , address=79.127.220.97:4361 , piece_id=LF7SIQCDAWI3PU3OEFZEKIATCQLSZDPDDPKAVL5J4QNW6FTVOMXA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900159643 (60h45m45.622722673s) [noise=true , address=104.36.82.56:4248 , piece_id=LBR27M3RL6HWYUGBVWOLMUWA72UZWPVXRFN4IMGTNWQVILOIC35A, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900383953 (60h45m45.622946993s) [noise=true , address=77.90.31.17:4780 , piece_id=RVVEQEN4J2CQTJMQ57TXH6JWJFD56ICZDLJLOBYV57SRHUYUFXGQ, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900413790 (60h45m45.62297683s) [noise=true , address=79.127.205.245:4228 , piece_id=6AW7WYWTBS24AP7U7EQKX2CRPOPEKUMM7IBTBMUH3WMZ7SWG26TQ, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900619403 (60h45m45.623182433s) [noise=true , address=79.127.205.246:4260 , piece_id=4I3LUF3EF46HFSDSVWFYR3ECRI6AGAGISDGZ7WAIRNZ43TJXP47Q, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.900851819 (60h45m45.623414859s) [noise=true , address=45.91.116.14:4194 , piece_id=GUGKMAK562MVIJY3RWZXEZ4YA2WJIMPD6R5YNYZOFE5A3REVGR7A, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901072160 (60h45m45.6236352s) [noise=true , address=77.90.31.13:4644 , piece_id=K4M3R5X54WGOQGOXV45GIVTTPBCOEA5VXBSSG5LGDUVRWG7UIJHA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901288794 (60h45m45.623851834s) [noise=true , address=104.36.82.46:4483 , piece_id=OLFZAE3K5IW673TRN3I3ICL7S54I5TXTYTLZAAPWMWMCHZAOI7RQ, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901516280 (60h45m45.62407932s) [noise=true , address=77.90.31.12:4603 , piece_id=UVPOPHV6OLJZ6EJA5VQJ24FWFWDXOZSP5TNLEWJSK2TYKVCIXVMA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901717444 (60h45m45.624280484s) [noise=true , address=79.127.205.233:4643 , piece_id=VAT2JO5OBUGJ4JYYNGLL55BNAAZX2YO3IJCCWTYEA23KVMOFIWTA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.901951663 (60h45m45.624514703s) [noise=true , address=79.127.205.235:4684 , piece_id=PVTG7I2PZK4EN7OV3UWN3TFBI3GDGCFESUMEPGQJZRG456HFIXAA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.902162616 (60h45m45.624725656s) [noise=true , address=79.127.219.38:4003 , piece_id=4X2JXWC5DSM4H2P4WWEDTVJC6WVK44UX7Y5ZBZJQO3KVB2SMLVZA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.902350776 (60h45m45.624913816s) [noise=true , address=77.90.31.11:4575 , piece_id=TNTM2XXQJ7SCCN4P3FLJ4CHGD3MJH7LJETQTECVIS3ATT4IGRDYA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:35.908037276 (60h45m45.630600316s) [noise=true , address=104.36.82.54:4177 , piece_id=EPC6HYPTCB2G6SGIEILYWCZIHAKNJY4CZS3CS4LPY4SFXJJOXMDA, seg_pos={0 0}, upload=58270]: Uploading piece... Jul 27 00:21:37.007965269 (60h45m46.730528309s) [upload=58270]: Next returned nil segment Jul 27 00:21:37.007989225 (60h45m46.730552265s) [upload=58270]: Waiting for error group managing segments... Jul 27 00:21:37.292932447 (60h45m47.015495497s) [noise=true , address=79.127.205.245:4228 , piece_id=6AW7WYWTBS24AP7U7EQKX2CRPOPEKUMM7IBTBMUH3WMZ7SWG26TQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:37.302353503 (60h45m47.024916543s) [noise=true , address=79.127.205.246:4260 , piece_id=4I3LUF3EF46HFSDSVWFYR3ECRI6AGAGISDGZ7WAIRNZ43TJXP47Q, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:37.318647989 (60h45m47.041211029s) [noise=true , address=79.127.205.235:4684 , piece_id=PVTG7I2PZK4EN7OV3UWN3TFBI3GDGCFESUMEPGQJZRG456HFIXAA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:37.551562895 (60h45m47.274125935s) [noise=true , address=79.127.205.233:4643 , piece_id=VAT2JO5OBUGJ4JYYNGLL55BNAAZX2YO3IJCCWTYEA23KVMOFIWTA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:38.094470500 (60h45m47.81703354s) [noise=true , address=79.127.219.38:4003 , piece_id=4X2JXWC5DSM4H2P4WWEDTVJC6WVK44UX7Y5ZBZJQO3KVB2SMLVZA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:40.968569045 (60h45m50.691132075s) [noise=true , address=79.127.220.97:4361 , piece_id=LF7SIQCDAWI3PU3OEFZEKIATCQLSZDPDDPKAVL5J4QNW6FTVOMXA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:42.570418614 (60h45m52.292981654s) [noise=true , address=77.90.31.13:4644 , piece_id=K4M3R5X54WGOQGOXV45GIVTTPBCOEA5VXBSSG5LGDUVRWG7UIJHA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:21:49.883461706 (60h45m59.606024746s) [noise=true , address=45.91.116.14:4194 , piece_id=GUGKMAK562MVIJY3RWZXEZ4YA2WJIMPD6R5YNYZOFE5A3REVGR7A, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:22:16.382783419 (60h46m26.105346449s) [noise=true , address=79.127.201.215:4347 , piece_id=RVPEXDTEBGLYUOFLW4SUWV3H3BRPWPCHSS5AOH6OHJFGHKPZLCUA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: <nil> Jul 27 00:22:34.796350709 (60h46m44.518913749s) [noise=true , address=77.90.31.11:4575 , piece_id=TNTM2XXQJ7SCCN4P3FLJ4CHGD3MJH7LJETQTECVIS3ATT4IGRDYA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 188.166.225.170:46826->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796387710 (60h46m44.51895074s) [noise=true , address=104.36.82.53:4123 , piece_id=5T64IZEVUHWPT3RU2XWC7C75FJPMJSXHGXIQV6HGWK3RGJA7DGJA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:104.36.82.53:4123): protocol: write tcp 188.166.225.170:60994->104.36.82.53:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796472282 (60h46m44.519035322s) [noise=true , address=104.36.82.54:4177 , piece_id=EPC6HYPTCB2G6SGIEILYWCZIHAKNJY4CZS3CS4LPY4SFXJJOXMDA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:39776->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796514873 (60h46m44.519077903s) [noise=true , address=104.36.82.41:4792 , piece_id=DO6QODRWE2DLNK4E4VQWMQ2NCUL4K5O2FZEL7TEBK7CR23L6RU3A, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 188.166.225.170:40470->104.36.82.41:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796546034 (60h46m44.519109074s) [noise=true , address=104.36.82.46:4483 , piece_id=OLFZAE3K5IW673TRN3I3ICL7S54I5TXTYTLZAAPWMWMCHZAOI7RQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: write tcp 188.166.225.170:42064->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796566903 (60h46m44.519129943s) [noise=true , address=77.90.31.10:4535 , piece_id=VCQHGHHFZRPMKE3IUQV5VHN5BTH6WXEE7GMLTDO65XDYOYM56WXQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 188.166.225.170:60080->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796622439 (60h46m44.519185479s) [noise=true , address=77.90.31.12:4603 , piece_id=UVPOPHV6OLJZ6EJA5VQJ24FWFWDXOZSP5TNLEWJSK2TYKVCIXVMA, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:22:34.796627359 (60h46m44.519190399s) [noise=true , address=104.36.82.56:4248 , piece_id=LBR27M3RL6HWYUGBVWOLMUWA72UZWPVXRFN4IMGTNWQVILOIC35A, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: write tcp 188.166.225.170:45436->104.36.82.56:4248: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:34.796683476 (60h46m44.519246517s) [noise=true , address=77.90.31.17:4780 , piece_id=RVVEQEN4J2CQTJMQ57TXH6JWJFD56ICZDLJLOBYV57SRHUYUFXGQ, seg_pos={0 0}, upload=58270]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: read tcp 188.166.225.170:43882->77.90.31.17:4780: use of closed network connection; context deadline exceeded Jul 27 00:22:34.796704736 (60h46m44.519267776s) [seg_pos={0 0}, upload=58270]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:22:35.024803809 (60h46m44.747366849s) [upload=58270]: Done uploading segments... Jul 27 00:22:35.024832904 (60h46m44.747395984s) [upload=58270]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.23053658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X......X...XX..XX.X.XX...X...............XX......X...] duration: 49.701526399s FAIL (2025-07-27 00:22:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.701539616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.088074195s FAIL (2025-07-27 00:22:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.088104712s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X........] duration: 7.374887328s FAIL (2025-07-27 00:22:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.37490891s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XX] duration: 30.232568957s FAIL (2025-07-27 00:22:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 5XA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; context deadline exceeded Jul 27 00:21:35.531548500 (60h45m45.25411154s) [seg_pos={0 0}, upload=58261]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:21:35.761627550 (60h45m45.48419059s) [upload=58261]: Done uploading segments... Jul 27 00:21:35.761652318 (60h45m45.484215358s) [upload=58261]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:21:36.634387963 (60h45m46.356950994s) [upload=58272]: Starting upload Jul 27 00:21:36.634444862 (60h45m46.357007902s) [upload=58272]: Done starting upload Jul 27 00:21:36.634605951 (60h45m46.357168991s) [upload=58272]: Uploading segments... Jul 27 00:21:36.634756889 (60h45m46.357319929s) [upload=58272]: Got next segment. Inline: false Jul 27 00:21:36.634822766 (60h45m46.357385806s) [upload=58272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:21:36.634829528 (60h45m46.357392568s) [upload=58272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:21:36.869871764 (60h45m46.592434804s) [seg_pos={0 0}, upload=58272]: Begin upload segment... Jul 27 00:21:36.869924796 (60h45m46.592487836s) [seg_pos={0 0}, upload=58272]: Done begin upload segment. Jul 27 00:21:36.869949984 (60h45m46.592513014s) [noise=true , address=77.90.31.15:4697 , piece_id=54FYZ7YHGMGG2FQ4RVKLOREX7VHCELNHZV4SGACPROVBOAJK4KNQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.869977898 (60h45m46.592540938s) [noise=true , address=104.36.82.7:4912 , piece_id=RHLYQUBM6TND3Y5DGYCVNCYGXSYBOSJROKFZQL7Z5D7HPJHQNE2Q, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870282761 (60h45m46.592845801s) [noise=true , address=45.91.116.10:4083 , piece_id=47RP7IMEY35X6WQSBEB2XFSFLNVTYGMEAEX4CVKRMFAR5W5EN4CQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870387622 (60h45m46.592950662s) [noise=true , address=45.91.116.14:4191 , piece_id=MFNRX6FLI4V6NHP6ICD2ID4QL2LCIKAAEDXJQZ23A6BXJI5JWFFA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870578717 (60h45m46.593141757s) [noise=true , address=45.91.116.13:4155 , piece_id=JKE3UKUHGFMKKTKHGJFELDPUZZANMVHYATNXP2CHELGRTKDRTUAQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.870883170 (60h45m46.59344621s) [noise=true , address=45.91.116.8:4045 , piece_id=6HIVPBWMWCEOEOLQBB2RTGTIGLVBQBUIO73Y556OJVO4PSFJHMEQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.871235334 (60h45m46.593798374s) [noise=true , address=45.91.116.11:4117 , piece_id=XK4RWDOPR26DDNV2TLN45S5FGMZZC74BWF6JD7XSTOF7RBV3TZEQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.871552140 (60h45m46.59411518s) [noise=true , address=77.90.31.13:4626 , piece_id=U6TKZLNAD2GOMJLKD6DFLRBD4AVTX66J5DTRYIZ4NXZX7ISDIJ6Q, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.871810995 (60h45m46.594374035s) [noise=true , address=45.91.116.9:4262 , piece_id=2B5LCT4Q44BHBVVAH53TZJ76IKCHLXI4EXZRMMOMLPK6K4D43PXA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.872097092 (60h45m46.594660122s) [noise=true , address=45.91.116.6:4973 , piece_id=FGJGJZORKSAQ3JPLC2EN7I76CBWQ6MZW4N3LJITNJEGGSIXY3M5A, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.872534589 (60h45m46.595097629s) [noise=true , address=104.36.82.15:4259 , piece_id=OUNXBRCJWK3FAPQ3RU6GXILFEACWBGVFE62OXABL4S5APDUAJ45A, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.872853119 (60h45m46.595416159s) [noise=true , address=45.91.116.5:4299 , piece_id=YI53TB6KHTXYVK3LETQJ2LWUHHT2GBXFWCNXZYKHQ2THQAWSAGYQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.873142062 (60h45m46.595705102s) [noise=true , address=77.90.31.12:4592 , piece_id=VDJ54S6BELCG7VFQ2IPBBEXVFWE23RM3N46PRK63W4JS3R2QOMWA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.873507962 (60h45m46.596071002s) [noise=true , address=77.90.31.11:4554 , piece_id=YLDQHAAN3MU6PL64MTQ33TXFH2LAKL6IVOFCFJIVPUIQ6D5A7RUQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.873766907 (60h45m46.596329947s) [noise=true , address=104.36.82.12:4150 , piece_id=KNB4EILZ4Z36S7REXMYE3V5BFRJN3LTSO5SSM2MQSQXKJK5RHZEQ, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:36.874007598 (60h45m46.596570638s) [noise=true , address=104.36.82.9:4006 , piece_id=A23OFTCZU2RYHBKVBFCYXSTH5RPASRQAY7K53EIZ44GYJ56APHTA, seg_pos={0 0}, upload=58272]: Uploading piece... Jul 27 00:21:38.600943896 (60h45m48.323506936s) [upload=58272]: Next returned nil segment Jul 27 00:21:38.600959396 (60h45m48.323522436s) [upload=58272]: Waiting for error group managing segments... Jul 27 00:21:38.890679217 (60h45m48.613242257s) [noise=true , address=45.91.116.5:4299 , piece_id=YI53TB6KHTXYVK3LETQJ2LWUHHT2GBXFWCNXZYKHQ2THQAWSAGYQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.892006296 (60h45m48.614569336s) [noise=true , address=45.91.116.10:4083 , piece_id=47RP7IMEY35X6WQSBEB2XFSFLNVTYGMEAEX4CVKRMFAR5W5EN4CQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.896651554 (60h45m48.619214584s) [noise=true , address=45.91.116.9:4262 , piece_id=2B5LCT4Q44BHBVVAH53TZJ76IKCHLXI4EXZRMMOMLPK6K4D43PXA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.912248305 (60h45m48.634811355s) [noise=true , address=45.91.116.11:4117 , piece_id=XK4RWDOPR26DDNV2TLN45S5FGMZZC74BWF6JD7XSTOF7RBV3TZEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.974286173 (60h45m48.696849213s) [noise=true , address=77.90.31.15:4697 , piece_id=54FYZ7YHGMGG2FQ4RVKLOREX7VHCELNHZV4SGACPROVBOAJK4KNQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.981026108 (60h45m48.703589148s) [noise=true , address=45.91.116.6:4973 , piece_id=FGJGJZORKSAQ3JPLC2EN7I76CBWQ6MZW4N3LJITNJEGGSIXY3M5A, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.984191114 (60h45m48.706754154s) [noise=true , address=45.91.116.13:4155 , piece_id=JKE3UKUHGFMKKTKHGJFELDPUZZANMVHYATNXP2CHELGRTKDRTUAQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:38.988692455 (60h45m48.711255495s) [noise=true , address=45.91.116.8:4045 , piece_id=6HIVPBWMWCEOEOLQBB2RTGTIGLVBQBUIO73Y556OJVO4PSFJHMEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:39.060708093 (60h45m48.783271133s) [noise=true , address=45.91.116.14:4191 , piece_id=MFNRX6FLI4V6NHP6ICD2ID4QL2LCIKAAEDXJQZ23A6BXJI5JWFFA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:40.276001385 (60h45m49.998564425s) [noise=true , address=77.90.31.13:4626 , piece_id=U6TKZLNAD2GOMJLKD6DFLRBD4AVTX66J5DTRYIZ4NXZX7ISDIJ6Q, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:21:40.495041585 (60h45m50.217604625s) [noise=true , address=77.90.31.12:4592 , piece_id=VDJ54S6BELCG7VFQ2IPBBEXVFWE23RM3N46PRK63W4JS3R2QOMWA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: <nil> Jul 27 00:22:05.994157902 (60h46m15.716720992s) [noise=true , address=104.36.82.9:4006 , piece_id=A23OFTCZU2RYHBKVBFCYXSTH5RPASRQAY7K53EIZ44GYJ56APHTA, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:44256->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994185896 (60h46m15.716748996s) [noise=true , address=104.36.82.15:4259 , piece_id=OUNXBRCJWK3FAPQ3RU6GXILFEACWBGVFE62OXABL4S5APDUAJ45A, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:50648->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994263364 (60h46m15.716826444s) [noise=true , address=77.90.31.11:4554 , piece_id=YLDQHAAN3MU6PL64MTQ33TXFH2LAKL6IVOFCFJIVPUIQ6D5A7RUQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:40988->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994301777 (60h46m15.716864907s) [noise=true , address=104.36.82.7:4912 , piece_id=RHLYQUBM6TND3Y5DGYCVNCYGXSYBOSJROKFZQL7Z5D7HPJHQNE2Q, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:50860->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:05.994355370 (60h46m15.71691841s) [noise=true , address=104.36.82.12:4150 , piece_id=KNB4EILZ4Z36S7REXMYE3V5BFRJN3LTSO5SSM2MQSQXKJK5RHZEQ, seg_pos={0 0}, upload=58272]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:22:05.994375438 (60h46m15.716938479s) [seg_pos={0 0}, upload=58272]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:22:06.225001157 (60h46m15.947564197s) [upload=58272]: Done uploading segments... Jul 27 00:22:06.225029662 (60h46m15.947592702s) [upload=58272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.232580891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X......X...XX..XX.X.XX...X...............XX......X..] duration: 45.764965656s FAIL (2025-07-27 00:21:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.764977398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.833947582s FAIL (2025-07-27 00:21:44 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.833976518s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXX] duration: 30.244238371s FAIL (2025-07-27 00:21:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.244263169s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..X] duration: 30.233843873s FAIL (2025-07-27 00:21:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== QZ2FJOCMHCR2UWYE5NWAIYSLA, seg_pos={0 0}, upload=58245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:52248->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:01.592669710 (60h44m11.31523275s) [seg_pos={0 0}, upload=58245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:20:01.592779350 (60h44m11.31534239s) [upload=58245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:20:01.592784700 (60h44m11.31534774s) [upload=58245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:20:01.869390209 (60h44m11.591953259s) [upload=58245]: Done uploading segments... Jul 27 00:20:01.869476946 (60h44m11.592039996s) [upload=58245]: Upload finished. err: <nil> Jul 27 00:21:06.175009756 (60h45m15.897572816s) [upload=58261]: Starting upload Jul 27 00:21:06.175141508 (60h45m15.897704578s) [upload=58261]: Done starting upload Jul 27 00:21:06.175279542 (60h45m15.897842592s) [upload=58261]: Uploading segments... Jul 27 00:21:06.175540240 (60h45m15.89810329s) [upload=58261]: Got next segment. Inline: false Jul 27 00:21:06.175585557 (60h45m15.898148597s) [upload=58261]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:21:06.175592631 (60h45m15.898155671s) [upload=58261]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:21:06.406959225 (60h45m16.129522265s) [seg_pos={0 0}, upload=58261]: Begin upload segment... Jul 27 00:21:06.407011886 (60h45m16.129574926s) [seg_pos={0 0}, upload=58261]: Done begin upload segment. Jul 27 00:21:06.407032184 (60h45m16.129595224s) [noise=true , address=104.36.82.6:4874 , piece_id=W5JXWM6LI2DOGEGNSBQKUILZDRWM4XOQUIER22ZZYCUYZG4RT5XA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407079575 (60h45m16.129642615s) [noise=true , address=77.90.31.14:4663 , piece_id=GCUT2KJEF2IOQLZCSNXMMADKWQTLPT4CXTBCZLRIIHEL5UPYXUJA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407387894 (60h45m16.129950934s) [noise=true , address=45.91.116.8:4048 , piece_id=KB2R5TXDK4FRYNAQA7ZW6DWXCTJZJZV57AKVTLCYJ2JA5MKONAQQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407602665 (60h45m16.130165716s) [noise=true , address=45.91.116.9:4261 , piece_id=OJLT3LDSHFRZUWXIXGTJ7BQOJRWV4IBUXQN447U2CCNRT7GQH7QQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407648203 (60h45m16.130211243s) [noise=true , address=45.91.116.12:4225 , piece_id=DGLBA4WLVH6OVUOFCV4Q5ZZA7FXWYHCRQOONNDVA7UZOBJH7HAJA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.407875728 (60h45m16.130438758s) [noise=true , address=45.91.116.7:4009 , piece_id=3BIFLZSWMA7UVCHTURVXTAZFOKXSTBVNW36EM2Z4OHYAA5ROMZTA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408041515 (60h45m16.130604555s) [noise=true , address=45.91.116.10:4083 , piece_id=G4SEWKI4SH6Z4RYNK4PBHFF4YZMXAZQFQP5ND4U75U2F6QGKRH2A, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408298907 (60h45m16.130861947s) [noise=true , address=77.90.31.21:4877 , piece_id=AWYRENUQ5CHGUMLIEH7PNQHGMOMA3IYBCIWF5VT72WOPDL5XKPFA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408400221 (60h45m16.130963262s) [noise=true , address=45.91.116.4:4940 , piece_id=J2MXG2QCOPIKHJ5UMOZ67HQZBE4YI35KREKJI4N2426DPP7DXTIQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.408811458 (60h45m16.131374498s) [noise=true , address=45.91.116.12:4226 , piece_id=HL2QW36BRZKVX3QWYG3XVPHSHA4ZRMOYR7CL3TSTRFJJM5UDEKWA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409030918 (60h45m16.131593958s) [noise=true , address=77.90.31.14:4662 , piece_id=L733LLJ5PPYJLRGNTP4AZYSTK5G65OA7SEVE5W3JTQNWBNOGWUKA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409055926 (60h45m16.131618976s) [noise=true , address=45.91.116.7:4010 , piece_id=CCKW4QLBBKSN23LTUYZE2GXEWB5WVHP4UHHOMDG6KNX4W7OI2OAQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409310784 (60h45m16.131873824s) [noise=true , address=45.91.116.4:4937 , piece_id=BDMQX3HLQC5HOAFJPQMMLRRQUQOYD36B7JTIRXNADQC3TR5FNXVA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409532317 (60h45m16.132095357s) [noise=true , address=77.90.31.12:4589 , piece_id=AZUC5LNR572P37DKJVMENLQZMPNSJG6CXRRU5E4KFDKQPHYVJ5DA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.409977360 (60h45m16.1325404s) [noise=true , address=104.36.82.14:4221 , piece_id=DOOTB4YCVPUSGVBEGGJINXGOXLFSOSTP2BCEMXNFTMCJOISTQRMA, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:06.410214352 (60h45m16.132777393s) [noise=true , address=104.36.82.3:4767 , piece_id=OP62ICVPR4YCHG7YU7BLQMMOSUROOA4N5GWYWHFYTLDTSVSKBEKQ, seg_pos={0 0}, upload=58261]: Uploading piece... Jul 27 00:21:08.132495033 (60h45m17.855058063s) [upload=58261]: Next returned nil segment Jul 27 00:21:08.132513439 (60h45m17.855076469s) [upload=58261]: Waiting for error group managing segments... Jul 27 00:21:08.423121589 (60h45m18.145684629s) [noise=true , address=45.91.116.4:4940 , piece_id=J2MXG2QCOPIKHJ5UMOZ67HQZBE4YI35KREKJI4N2426DPP7DXTIQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.438822710 (60h45m18.16138575s) [noise=true , address=45.91.116.12:4226 , piece_id=HL2QW36BRZKVX3QWYG3XVPHSHA4ZRMOYR7CL3TSTRFJJM5UDEKWA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.447263099 (60h45m18.169826139s) [noise=true , address=45.91.116.7:4009 , piece_id=3BIFLZSWMA7UVCHTURVXTAZFOKXSTBVNW36EM2Z4OHYAA5ROMZTA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.510942547 (60h45m18.233505587s) [noise=true , address=45.91.116.12:4225 , piece_id=DGLBA4WLVH6OVUOFCV4Q5ZZA7FXWYHCRQOONNDVA7UZOBJH7HAJA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.517049822 (60h45m18.239612862s) [noise=true , address=77.90.31.14:4662 , piece_id=L733LLJ5PPYJLRGNTP4AZYSTK5G65OA7SEVE5W3JTQNWBNOGWUKA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.519177914 (60h45m18.241740954s) [noise=true , address=77.90.31.14:4663 , piece_id=GCUT2KJEF2IOQLZCSNXMMADKWQTLPT4CXTBCZLRIIHEL5UPYXUJA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.521029547 (60h45m18.243592587s) [noise=true , address=45.91.116.9:4261 , piece_id=OJLT3LDSHFRZUWXIXGTJ7BQOJRWV4IBUXQN447U2CCNRT7GQH7QQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.531154458 (60h45m18.253717488s) [noise=true , address=45.91.116.10:4083 , piece_id=G4SEWKI4SH6Z4RYNK4PBHFF4YZMXAZQFQP5ND4U75U2F6QGKRH2A, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.531497534 (60h45m18.254060574s) [noise=true , address=45.91.116.7:4010 , piece_id=CCKW4QLBBKSN23LTUYZE2GXEWB5WVHP4UHHOMDG6KNX4W7OI2OAQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.573165027 (60h45m18.295728067s) [noise=true , address=45.91.116.8:4048 , piece_id=KB2R5TXDK4FRYNAQA7ZW6DWXCTJZJZV57AKVTLCYJ2JA5MKONAQQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:08.651239016 (60h45m18.373802056s) [noise=true , address=45.91.116.4:4937 , piece_id=BDMQX3HLQC5HOAFJPQMMLRRQUQOYD36B7JTIRXNADQC3TR5FNXVA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:09.854505436 (60h45m19.577068476s) [noise=true , address=77.90.31.12:4589 , piece_id=AZUC5LNR572P37DKJVMENLQZMPNSJG6CXRRU5E4KFDKQPHYVJ5DA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: <nil> Jul 27 00:21:35.531229941 (60h45m45.253792981s) [noise=true , address=77.90.31.21:4877 , piece_id=AWYRENUQ5CHGUMLIEH7PNQHGMOMA3IYBCIWF5VT72WOPDL5XKPFA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:35.531306297 (60h45m45.253869337s) [noise=true , address=104.36.82.14:4221 , piece_id=DOOTB4YCVPUSGVBEGGJINXGOXLFSOSTP2BCEMXNFTMCJOISTQRMA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:46220->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:35.531478837 (60h45m45.254041867s) [noise=true , address=104.36.82.3:4767 , piece_id=OP62ICVPR4YCHG7YU7BLQMMOSUROOA4N5GWYWHFYTLDTSVSKBEKQ, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:56604->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:35.531513833 (60h45m45.254076874s) [noise=true , address=104.36.82.6:4874 , piece_id=W5JXWM6LI2DOGEGNSBQKUILZDRWM4XOQUIER22ZZYCUYZG4RT5XA, seg_pos={0 0}, upload=58261]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; context deadline exceeded Jul 27 00:21:35.531548500 (60h45m45.25411154s) [seg_pos={0 0}, upload=58261]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:21:35.761627550 (60h45m45.48419059s) [upload=58261]: Done uploading segments... Jul 27 00:21:35.761652318 (60h45m45.484215358s) [upload=58261]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233871567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238876021s FAIL (2025-07-27 00:21:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 424571561 (60h44m45.147134601s) [seg_pos={0 0}, upload=58255]: Done begin upload segment. Jul 27 00:20:35.424594505 (60h44m45.147157545s) [noise=true , address=104.36.82.7:4940 , piece_id=UKEUF6RRS4PWZXZWUQ7J3QCF3QBGUMGNJDV22CWTTKCXRIKG7ZVA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.424630664 (60h44m45.147193724s) [noise=true , address=77.90.31.11:4585 , piece_id=WKRFEUME7524PTG4IIONOF7ZXEGGVSWQYOEHOPT6OKSIDCJO6HGQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.424921541 (60h44m45.147484581s) [noise=true , address=79.127.220.101:4041 , piece_id=LTQQ23IJR7L4F5BHXXTXECVESMZVWTJVVPEXS6ZCAUHU6SIVPGEQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425312298 (60h44m45.147875338s) [noise=true , address=104.36.82.54:4174 , piece_id=O3NFUA65AIHQ5HGPFD6QIWJE2B67LCN7IF4KOJXPRUMZBM2AUGQQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425338829 (60h44m45.147901869s) [noise=true , address=45.91.116.9:4277 , piece_id=ITERKIQR4HPWVCQBABQD2WBLCQUMYONHI4BRQMI5LB2F3VBZQSAA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425619416 (60h44m45.148182446s) [noise=true , address=77.90.31.16:4743 , piece_id=RSPTFHDLZFQX32IIVTES3FQJXH4CVBHEJM763ESVM4P5GDDH7YWQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425623674 (60h44m45.148186714s) [noise=true , address=79.127.205.234:4618 , piece_id=PHHIBKQQ5VOOZQPPDYR5GHHBYA5LO6S46A33WVZAAXMC54SFRK4A, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425904220 (60h44m45.14846726s) [noise=true , address=77.90.31.18:4809 , piece_id=FR5PFVGFGDDG62ZYSXRUFZ3G5PAELFME62MSCDD5QNVXYPZ6G5KA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.425975788 (60h44m45.148538828s) [noise=true , address=79.127.219.38:4018 , piece_id=7CLUGVYZRHVQXU3FPZPTGHXPATICXOC577HINQLQZXPBUYMGVWMA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426166362 (60h44m45.148729402s) [noise=true , address=64.49.13.138:4482 , piece_id=4FLURCOOV7NOQRRRQ4ETC4KLULNRDSLCUOQHDSGJXDER276237EQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426234893 (60h44m45.148797923s) [noise=true , address=77.90.31.15:4731 , piece_id=D23WPZMNCENQD3W23UMASXZP7O5XUV4NLQSPKVG2GRDETAN72ZSA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426363039 (60h44m45.148926079s) [noise=true , address=104.36.82.56:4262 , piece_id=XRW7RQNBUC3ANMKGMMH7NFDHRHIMX6M6QRXGANIEZSXCCH2NFPEQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426432210 (60h44m45.14899525s) [noise=true , address=77.90.31.13:4638 , piece_id=F53EJRYARYEUVA4WET2MK3737YE7H7EOPMECE7DCZWO2GD736R3Q, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426557420 (60h44m45.14912046s) [noise=true , address=104.36.82.47:4523 , piece_id=MJCYOBRC2QBBKSHCIC5MBPSFLLFE4VIPGBFJ25ZKRZ5BZY4A4GOQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426672901 (60h44m45.149235961s) [noise=true , address=79.127.205.246:4260 , piece_id=RCJ2F55PR5ETM5JUCCGJMOHBUQ2TA53SU645RJNQKD5CDSWQMKOA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.426754527 (60h44m45.149317567s) [noise=true , address=104.36.82.46:4516 , piece_id=WQX2LHBHHU3O4LKULCG2ZSU77SDTQOXVODBWADVFRY2NZZGJ5QAQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.427151387 (60h44m45.149714427s) [noise=true , address=77.90.31.17:4790 , piece_id=JYEZQOD2BU7PQMW3F2D6WGO626K45OSWRUVZRA37EPZJCRNMOWWQ, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:35.427895691 (60h44m45.150458731s) [noise=true , address=79.127.201.215:4346 , piece_id=NPAKMSYWTBMKKJYHMWBRX7FHA4VYW6MFJM6OZAWA26W26KV6ULLA, seg_pos={0 0}, upload=58255]: Uploading piece... Jul 27 00:20:36.541327631 (60h44m46.263890671s) [upload=58255]: Next returned nil segment Jul 27 00:20:36.541345114 (60h44m46.263908144s) [upload=58255]: Waiting for error group managing segments... Jul 27 00:20:36.549751749 (60h44m46.272314779s) [noise=true , address=77.90.31.13:4638 , piece_id=F53EJRYARYEUVA4WET2MK3737YE7H7EOPMECE7DCZWO2GD736R3Q, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: storage node overloaded, request limit: 2000 Jul 27 00:20:36.840745377 (60h44m46.563308417s) [noise=true , address=79.127.205.234:4618 , piece_id=PHHIBKQQ5VOOZQPPDYR5GHHBYA5LO6S46A33WVZAAXMC54SFRK4A, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:36.852153805 (60h44m46.574716845s) [noise=true , address=79.127.205.246:4260 , piece_id=RCJ2F55PR5ETM5JUCCGJMOHBUQ2TA53SU645RJNQKD5CDSWQMKOA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:37.314951215 (60h44m47.037514255s) [noise=true , address=79.127.201.215:4346 , piece_id=NPAKMSYWTBMKKJYHMWBRX7FHA4VYW6MFJM6OZAWA26W26KV6ULLA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:37.336180925 (60h44m47.058743955s) [noise=true , address=79.127.219.38:4018 , piece_id=7CLUGVYZRHVQXU3FPZPTGHXPATICXOC577HINQLQZXPBUYMGVWMA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:37.352515446 (60h44m47.075078486s) [noise=true , address=77.90.31.18:4809 , piece_id=FR5PFVGFGDDG62ZYSXRUFZ3G5PAELFME62MSCDD5QNVXYPZ6G5KA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:38.143791747 (60h44m47.866354787s) [noise=true , address=77.90.31.16:4743 , piece_id=RSPTFHDLZFQX32IIVTES3FQJXH4CVBHEJM763ESVM4P5GDDH7YWQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:39.152402764 (60h44m48.874965804s) [noise=true , address=64.49.13.138:4482 , piece_id=4FLURCOOV7NOQRRRQ4ETC4KLULNRDSLCUOQHDSGJXDER276237EQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:40.836663303 (60h44m50.559226333s) [noise=true , address=45.91.116.9:4277 , piece_id=ITERKIQR4HPWVCQBABQD2WBLCQUMYONHI4BRQMI5LB2F3VBZQSAA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:43.987867722 (60h44m53.710430782s) [noise=true , address=79.127.220.101:4041 , piece_id=LTQQ23IJR7L4F5BHXXTXECVESMZVWTJVVPEXS6ZCAUHU6SIVPGEQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:20:57.821042376 (60h45m7.543605416s) [noise=true , address=77.90.31.15:4731 , piece_id=D23WPZMNCENQD3W23UMASXZP7O5XUV4NLQSPKVG2GRDETAN72ZSA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: <nil> Jul 27 00:21:34.333503139 (60h45m44.056066179s) [noise=true , address=77.90.31.11:4585 , piece_id=WKRFEUME7524PTG4IIONOF7ZXEGGVSWQYOEHOPT6OKSIDCJO6HGQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: rpc: dial tcp 77.90.31.11:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333545630 (60h45m44.05610867s) [noise=true , address=104.36.82.46:4516 , piece_id=WQX2LHBHHU3O4LKULCG2ZSU77SDTQOXVODBWADVFRY2NZZGJ5QAQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:12FRYABJbf6iHTm3WiYcL192kFth6rHjrcMAS3ahF8ddm4ox6Uj, address:104.36.82.46:4516): protocol: rpc: dial tcp 104.36.82.46:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333577962 (60h45m44.056141002s) [noise=true , address=104.36.82.56:4262 , piece_id=XRW7RQNBUC3ANMKGMMH7NFDHRHIMX6M6QRXGANIEZSXCCH2NFPEQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333607038 (60h45m44.056170078s) [noise=true , address=104.36.82.47:4523 , piece_id=MJCYOBRC2QBBKSHCIC5MBPSFLLFE4VIPGBFJ25ZKRZ5BZY4A4GOQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 188.166.225.170:56130->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333647405 (60h45m44.056210446s) [noise=true , address=77.90.31.17:4790 , piece_id=JYEZQOD2BU7PQMW3F2D6WGO626K45OSWRUVZRA37EPZJCRNMOWWQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:21:34.333651112 (60h45m44.056214152s) [noise=true , address=104.36.82.54:4174 , piece_id=O3NFUA65AIHQ5HGPFD6QIWJE2B67LCN7IF4KOJXPRUMZBM2AUGQQ, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333658977 (60h45m44.056222017s) [noise=true , address=104.36.82.7:4940 , piece_id=UKEUF6RRS4PWZXZWUQ7J3QCF3QBGUMGNJDV22CWTTKCXRIKG7ZVA, seg_pos={0 0}, upload=58255]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:104.36.82.7:4940): protocol: rpc: dial tcp 104.36.82.7:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:34.333672162 (60h45m44.056235202s) [seg_pos={0 0}, upload=58255]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:21:34.570295584 (60h45m44.292858624s) [upload=58255]: Done uploading segments... Jul 27 00:21:34.570325080 (60h45m44.29288812s) [upload=58255]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.238888265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.826511986s FAIL (2025-07-27 00:21:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.826537125s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.......] duration: 7.463890186s FAIL (2025-07-27 00:21:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.463907958s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X..X.....X......X...XX..XX.X.XX...X...............XX......X.] duration: 56.623183294s FAIL (2025-07-27 00:20:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.623195779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X......] duration: 6.369788097s FAIL (2025-07-27 00:20:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.369810501s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23523877s FAIL (2025-07-27 00:20:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 7293336s) [noise=true , address=79.127.219.48:4074 , piece_id=LEZQ4DDHMZVGUJR5IDAL2QUPYIYADFDA3DKCGGOEQBZY3LRYH4OA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.954813295 (60h43m44.677376325s) [noise=true , address=77.90.31.11:4575 , piece_id=UBD3K5OFOLEGY52A7Q47G3ZAJ3J56RDMUBX7SUD4HOQO5TP5AE4A, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955085847 (60h43m44.677648887s) [noise=true , address=104.36.82.46:4493 , piece_id=PDNBKD2BHCATEJNK52V2J5WZHW3PVJCP2VUXS3ZIJ6CPPU3H4TAA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955335805 (60h43m44.677898845s) [noise=true , address=64.49.13.137:4502 , piece_id=OQWQP3TEDSZ2K3U2AJRL462WZEXBKBZBM2MJUFKKTFBETCNW4IDQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955583218 (60h43m44.678146258s) [noise=true , address=79.127.219.38:4006 , piece_id=CDHYTG33QZLWVGX6ISIZUHM6XR3QFCWSOOPTZUVGLL43BDMWI3IA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.955821484 (60h43m44.678384524s) [noise=true , address=104.36.82.54:4163 , piece_id=HN35V7VNNRR2TGIDM5PLXPDMKLR2ZFYO2TZI4NDSFBLFWR7J5AAQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956036766 (60h43m44.678599806s) [noise=true , address=77.90.31.18:4823 , piece_id=FZLULHRAMEYKJ254U5XWAUWV7RJFHSVEX4DLVJ3CGJDSUCNFEJKA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956194448 (60h43m44.678757478s) [noise=true , address=79.127.205.245:4245 , piece_id=CJM7ISA7PFNHW2D7XRGRKVDZYTJC7IDH5DDX3BAANI7ZNHXRSU4Q, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956442703 (60h43m44.679005733s) [noise=true , address=104.36.82.47:4537 , piece_id=USZNOO2ROXDEX4ZOLHL27OBJKZFUYY2WDEA6BN44UW4WNIPE4SGQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956666812 (60h43m44.679229852s) [noise=true , address=45.91.116.8:4062 , piece_id=LN7P5JGQMFS5LOSDCF7OO4RPODA76C2ZCFBTWAVMDTU3GRN54KRQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956709563 (60h43m44.679272603s) [noise=true , address=79.127.205.246:4282 , piece_id=T2AIQLDZUY27WAHBUNBPBN3I2DPY3UVWBT3BLCOWC5NDDNBZYPSQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.956923593 (60h43m44.679486633s) [noise=true , address=104.36.82.11:4116 , piece_id=V2722NTPPIN5C4CCXEEXRKNDUAKTVSC7DJF5N4SDSGWNJCPM26JA, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957114127 (60h43m44.679677168s) [noise=true , address=104.36.82.55:4202 , piece_id=4QG5MNIRYMX5DYQ72D2CH5ZTZDUO2NTIEDTNLYJ6ZJJX6NDWRDFQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957265508 (60h43m44.679828548s) [noise=true , address=79.127.201.214:4321 , piece_id=M62AFBCW435H6PQEWEALBJKZU6NGQPETTDI6Y6KGE5JJECOLW77Q, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957479977 (60h43m44.680043017s) [noise=true , address=79.127.220.97:4361 , piece_id=O52ZWST27VEH2JQYT5BGK6ZRPDZI6ZB4EJDQNHG2OXZU5EM4OPWQ, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957562455 (60h43m44.680125485s) [noise=true , address=104.36.82.52:4096 , piece_id=IQRCWGOOFGD3TFDPE6JGEJQZTY372EGUOWFXUAYB265G2EROTB5A, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:34.957750064 (60h43m44.680313094s) [noise=true , address=104.36.82.14:4244 , piece_id=VM5UEPCKLGDUF22ZSFHVI3OMJEYKI4LCBJIQYSCSEI6Z2XG4D77A, seg_pos={0 0}, upload=58240]: Uploading piece... Jul 27 00:19:36.073032685 (60h43m45.795595725s) [upload=58240]: Next returned nil segment Jul 27 00:19:36.073050499 (60h43m45.795613529s) [upload=58240]: Waiting for error group managing segments... Jul 27 00:19:36.396390456 (60h43m46.118953496s) [noise=true , address=79.127.205.245:4245 , piece_id=CJM7ISA7PFNHW2D7XRGRKVDZYTJC7IDH5DDX3BAANI7ZNHXRSU4Q, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.481879673 (60h43m46.204442703s) [noise=true , address=64.49.13.137:4502 , piece_id=OQWQP3TEDSZ2K3U2AJRL462WZEXBKBZBM2MJUFKKTFBETCNW4IDQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.765119535 (60h43m46.487682575s) [noise=true , address=79.127.220.97:4361 , piece_id=O52ZWST27VEH2JQYT5BGK6ZRPDZI6ZB4EJDQNHG2OXZU5EM4OPWQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.859649860 (60h43m46.58221289s) [noise=true , address=79.127.205.246:4282 , piece_id=T2AIQLDZUY27WAHBUNBPBN3I2DPY3UVWBT3BLCOWC5NDDNBZYPSQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.947368111 (60h43m46.669931152s) [noise=true , address=79.127.219.48:4074 , piece_id=LEZQ4DDHMZVGUJR5IDAL2QUPYIYADFDA3DKCGGOEQBZY3LRYH4OA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:36.958322321 (60h43m46.680885361s) [noise=true , address=79.127.219.38:4006 , piece_id=CDHYTG33QZLWVGX6ISIZUHM6XR3QFCWSOOPTZUVGLL43BDMWI3IA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:37.085849138 (60h43m46.808412178s) [noise=true , address=77.90.31.18:4823 , piece_id=FZLULHRAMEYKJ254U5XWAUWV7RJFHSVEX4DLVJ3CGJDSUCNFEJKA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:38.455975221 (60h43m48.178538251s) [noise=true , address=79.127.201.214:4321 , piece_id=M62AFBCW435H6PQEWEALBJKZU6NGQPETTDI6Y6KGE5JJECOLW77Q, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:19:57.616809053 (60h44m7.339372093s) [noise=true , address=45.91.116.8:4062 , piece_id=LN7P5JGQMFS5LOSDCF7OO4RPODA76C2ZCFBTWAVMDTU3GRN54KRQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: <nil> Jul 27 00:20:33.848344606 (60h44m43.570907646s) [noise=true , address=104.36.82.46:4493 , piece_id=PDNBKD2BHCATEJNK52V2J5WZHW3PVJCP2VUXS3ZIJ6CPPU3H4TAA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848424269 (60h44m43.570987309s) [noise=true , address=104.36.82.54:4163 , piece_id=HN35V7VNNRR2TGIDM5PLXPDMKLR2ZFYO2TZI4NDSFBLFWR7J5AAQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848478953 (60h44m43.571042004s) [noise=true , address=104.36.82.52:4096 , piece_id=IQRCWGOOFGD3TFDPE6JGEJQZTY372EGUOWFXUAYB265G2EROTB5A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: write tcp 188.166.225.170:50222->104.36.82.52:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848592030 (60h44m43.57115507s) [noise=true , address=77.90.31.17:4786 , piece_id=RJ5FD7P7HCVP3SQZKWARZCFZFTNCCD25DNUXNHCVB2CQOUWSRO6A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: rpc: dial tcp 77.90.31.17:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848641575 (60h44m43.571204615s) [noise=true , address=77.90.31.11:4575 , piece_id=UBD3K5OFOLEGY52A7Q47G3ZAJ3J56RDMUBX7SUD4HOQO5TP5AE4A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848687993 (60h44m43.571251033s) [noise=true , address=104.36.82.11:4116 , piece_id=V2722NTPPIN5C4CCXEEXRKNDUAKTVSC7DJF5N4SDSGWNJCPM26JA, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:104.36.82.11:4116): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:33.848698293 (60h44m43.571261333s) [noise=true , address=104.36.82.47:4537 , piece_id=USZNOO2ROXDEX4ZOLHL27OBJKZFUYY2WDEA6BN44UW4WNIPE4SGQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848757005 (60h44m43.571320045s) [noise=true , address=104.36.82.55:4202 , piece_id=4QG5MNIRYMX5DYQ72D2CH5ZTZDUO2NTIEDTNLYJ6ZJJX6NDWRDFQ, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848772515 (60h44m43.571335566s) [noise=true , address=104.36.82.14:4244 , piece_id=VM5UEPCKLGDUF22ZSFHVI3OMJEYKI4LCBJIQYSCSEI6Z2XG4D77A, seg_pos={0 0}, upload=58240]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:4244): protocol: write tcp 188.166.225.170:53038->104.36.82.14:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:33.848793856 (60h44m43.571356896s) [seg_pos={0 0}, upload=58240]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:20:34.081476006 (60h44m43.804039046s) [upload=58240]: Done uploading segments... Jul 27 00:20:34.081502797 (60h44m43.804065837s) [upload=58240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.23525465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXX] duration: 30.243351076s FAIL (2025-07-27 00:20:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.243363519s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.110352396s FAIL (2025-07-27 00:20:20 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.110378826s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.....] duration: 7.773077094s FAIL (2025-07-27 00:20:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.773097984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..] duration: 6.812501964s FAIL (2025-07-27 00:20:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:18:24.334097790 (60h42m34.05666082s) [upload=58224]: Starting upload Jul 27 00:18:24.334154699 (60h42m34.056717779s) [upload=58224]: Done starting upload Jul 27 00:18:24.334237908 (60h42m34.056800938s) [upload=58224]: Uploading segments... Jul 27 00:18:24.334437640 (60h42m34.05700067s) [upload=58224]: Got next segment. Inline: false Jul 27 00:18:24.334483638 (60h42m34.057046679s) [upload=58224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:24.334488578 (60h42m34.057051618s) [upload=58224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:24.564881187 (60h42m34.287444227s) [seg_pos={0 0}, upload=58224]: Begin upload segment... Jul 27 00:18:24.564929989 (60h42m34.28749303s) [seg_pos={0 0}, upload=58224]: Done begin upload segment. Jul 27 00:18:24.564954016 (60h42m34.287517056s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.564998090 (60h42m34.28756112s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565311169 (60h42m34.287874209s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565454935 (60h42m34.288017965s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565559765 (60h42m34.288122805s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565729580 (60h42m34.28829262s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565840673 (60h42m34.288403713s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566026167 (60h42m34.288589207s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566132010 (60h42m34.28869505s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566382689 (60h42m34.288945719s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566620755 (60h42m34.289183795s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566845164 (60h42m34.289408204s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567029938 (60h42m34.289592978s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567212567 (60h42m34.289775607s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567393123 (60h42m34.289956163s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567580712 (60h42m34.290143742s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:26.259818721 (60h42m35.982381751s) [upload=58224]: Next returned nil segment Jul 27 00:18:26.259835624 (60h42m35.982398654s) [upload=58224]: Waiting for error group managing segments... Jul 27 00:18:26.584579627 (60h42m36.307142667s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592038658 (60h42m36.314601688s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592821997 (60h42m36.315385038s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.662347510 (60h42m36.38491055s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.679168664 (60h42m36.401731704s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.700554562 (60h42m36.423117592s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.736485559 (60h42m36.459048589s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:27.128265201 (60h42m36.850828241s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.113985780 (60h42m37.83654882s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.206936645 (60h42m37.929499685s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:53.685476775 (60h43m3.408039815s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685723547 (60h43m3.408286577s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:59780->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685745219 (60h43m3.408308249s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685781728 (60h43m3.408344768s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685849078 (60h43m3.408412118s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685877802 (60h43m3.408440842s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685897851 (60h43m3.408460891s) [seg_pos={0 0}, upload=58224]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:53.911235189 (60h43m3.633798229s) [upload=58224]: Done uploading segments... Jul 27 00:18:53.911264825 (60h43m3.633827865s) [upload=58224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.812526923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XX..X.....X......X...XX..XX.X.XX...X...............XX......X] duration: 1m0.001500679s FAIL (2025-07-27 00:20:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001526397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XX] duration: 30.231894723s FAIL (2025-07-27 00:19:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.231912506s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX.] duration: 7.163360941s FAIL (2025-07-27 00:19:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:18:24.334097790 (60h42m34.05666082s) [upload=58224]: Starting upload Jul 27 00:18:24.334154699 (60h42m34.056717779s) [upload=58224]: Done starting upload Jul 27 00:18:24.334237908 (60h42m34.056800938s) [upload=58224]: Uploading segments... Jul 27 00:18:24.334437640 (60h42m34.05700067s) [upload=58224]: Got next segment. Inline: false Jul 27 00:18:24.334483638 (60h42m34.057046679s) [upload=58224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:24.334488578 (60h42m34.057051618s) [upload=58224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:24.564881187 (60h42m34.287444227s) [seg_pos={0 0}, upload=58224]: Begin upload segment... Jul 27 00:18:24.564929989 (60h42m34.28749303s) [seg_pos={0 0}, upload=58224]: Done begin upload segment. Jul 27 00:18:24.564954016 (60h42m34.287517056s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.564998090 (60h42m34.28756112s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565311169 (60h42m34.287874209s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565454935 (60h42m34.288017965s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565559765 (60h42m34.288122805s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565729580 (60h42m34.28829262s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565840673 (60h42m34.288403713s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566026167 (60h42m34.288589207s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566132010 (60h42m34.28869505s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566382689 (60h42m34.288945719s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566620755 (60h42m34.289183795s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566845164 (60h42m34.289408204s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567029938 (60h42m34.289592978s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567212567 (60h42m34.289775607s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567393123 (60h42m34.289956163s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567580712 (60h42m34.290143742s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:26.259818721 (60h42m35.982381751s) [upload=58224]: Next returned nil segment Jul 27 00:18:26.259835624 (60h42m35.982398654s) [upload=58224]: Waiting for error group managing segments... Jul 27 00:18:26.584579627 (60h42m36.307142667s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592038658 (60h42m36.314601688s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592821997 (60h42m36.315385038s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.662347510 (60h42m36.38491055s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.679168664 (60h42m36.401731704s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.700554562 (60h42m36.423117592s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.736485559 (60h42m36.459048589s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:27.128265201 (60h42m36.850828241s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.113985780 (60h42m37.83654882s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.206936645 (60h42m37.929499685s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:53.685476775 (60h43m3.408039815s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685723547 (60h43m3.408286577s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:59780->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685745219 (60h43m3.408308249s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685781728 (60h43m3.408344768s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685849078 (60h43m3.408412118s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685877802 (60h43m3.408440842s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685897851 (60h43m3.408460891s) [seg_pos={0 0}, upload=58224]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:53.911235189 (60h43m3.633798229s) [upload=58224]: Done uploading segments... Jul 27 00:18:53.911264825 (60h43m3.633827865s) [upload=58224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.163405718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234351135s FAIL (2025-07-27 00:19:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece_id=LUEEX3QN6AM2FT75GANCNOS6PQGZCZWKH4F3VCH24LPNFT56W4CA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.481894408 (60h42m44.204457448s) [noise=true , address=104.36.82.46:4501 , piece_id=GJYUQA7AFKYED4N7H2ONXWTD4SQ7OLQTTVVHQDRQ2OV7HPP37QWQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.482222586 (60h42m44.204785626s) [noise=true , address=79.127.205.234:4607 , piece_id=DWAZDYUI74PLSMOZKHJ3MFRC65NFWIYRIACVPKLZZRLXVBKSYJRA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.482581452 (60h42m44.205144482s) [noise=true , address=77.90.31.19:4871 , piece_id=4XEVC563OQI7YCCQ2JRDR66NKM5RORT2A2ZPOWD65WZJ5QSEC2ZQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.482847722 (60h42m44.205410762s) [noise=true , address=77.90.31.17:4778 , piece_id=QIYIIBCZGQPNNOVHC7ZNF76VAACNJGMXTCO5EQUJZE4DNELQH6IQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483105084 (60h42m44.205668124s) [noise=true , address=104.36.82.55:4194 , piece_id=HNEE5JZ6BZ5JHGROQSSKDSBIGUMRLGDVLJJ6GSYA4KKIMBRVLKQQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483358819 (60h42m44.205921859s) [noise=true , address=64.49.13.138:4488 , piece_id=TE7I4CJMA767HHH5YKWMLJOE5LS5M32NGI76X62WQBO2G5NHGM4Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483622043 (60h42m44.206185083s) [noise=true , address=77.90.31.16:4751 , piece_id=LJB7N2VOHYA43XZP32I4RPHRI4G63IWF33JVFBIQY56YFYIEAU2A, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.483821244 (60h42m44.206384284s) [noise=true , address=77.90.31.18:4817 , piece_id=LAFPCQDUFJCZQQRGIIIMGKT3RY527BYIN55QPS2KGF4ADAWLI3UA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484083887 (60h42m44.206646927s) [noise=true , address=104.36.82.54:4190 , piece_id=SPKGL26ZFVI6VWJQIGBOG3BZZ2Z4RTPHUGLDQUIKZRWTEFRG463Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484286775 (60h42m44.206849815s) [noise=true , address=45.91.116.9:4277 , piece_id=KO52SDQPK4MCQD7XAF7Q5ZCIKAGBVRGFPWKXZMW45C4TDWDIM56Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484480996 (60h42m44.207044036s) [noise=true , address=104.36.82.47:4546 , piece_id=44H6TXDATOOSKBWLNRBW6D77CUT4RBDONMXI34UOEAUGLQCBBJTQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484665229 (60h42m44.207228259s) [noise=true , address=79.127.219.38:4001 , piece_id=KMTLQ2CLCS6ZF66SF7JSPUPPIWZVQFXYPCKDS3KIDDMPVJFLDUEA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.484863058 (60h42m44.207426098s) [noise=true , address=104.36.82.52:4119 , piece_id=27ZPC7G2IMJJ6GQWD3WZGE7RFZKPXKOSPUE7HO2SQHDW3KIBSR5Q, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485056027 (60h42m44.207619057s) [noise=true , address=79.127.205.246:4283 , piece_id=SVLSGY6XYJFPIBQLBHBQFZTCKLAINYZABS5PKGCL7TC3CJM7Q6RQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485242083 (60h42m44.207805123s) [noise=true , address=77.90.31.11:4576 , piece_id=7ZW7TOSVB5BPFZT4HGNCBBRCTESX5GNM5V5ULBLYU4MYUFAVZEJQ, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485441644 (60h42m44.208004685s) [noise=true , address=104.36.82.53:4147 , piece_id=COCCHCNFRMA425SNFZLKTZNLCXYEEATE3IWP4KERGKJ72VL47BNA, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:34.485668378 (60h42m44.208231408s) [noise=true , address=77.90.31.12:4611 , piece_id=JTFTDCB2KJ3WJ75XDXUYGEN55QL4WKCKPG2TW5TE7FGT3TLQFO6A, seg_pos={0 0}, upload=58226]: Uploading piece... Jul 27 00:18:35.510716284 (60h42m45.233279324s) [noise=true , address=77.90.31.19:4871 , piece_id=4XEVC563OQI7YCCQ2JRDR66NKM5RORT2A2ZPOWD65WZJ5QSEC2ZQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 27 00:18:35.605175292 (60h42m45.327738322s) [upload=58226]: Next returned nil segment Jul 27 00:18:35.605190971 (60h42m45.327754011s) [upload=58226]: Waiting for error group managing segments... Jul 27 00:18:35.887562934 (60h42m45.610125974s) [noise=true , address=79.127.205.246:4283 , piece_id=SVLSGY6XYJFPIBQLBHBQFZTCKLAINYZABS5PKGCL7TC3CJM7Q6RQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:35.895500381 (60h42m45.618063421s) [noise=true , address=79.127.205.233:4650 , piece_id=LUEEX3QN6AM2FT75GANCNOS6PQGZCZWKH4F3VCH24LPNFT56W4CA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:35.904784985 (60h42m45.627348015s) [noise=true , address=79.127.205.234:4607 , piece_id=DWAZDYUI74PLSMOZKHJ3MFRC65NFWIYRIACVPKLZZRLXVBKSYJRA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:36.392178315 (60h42m46.114741355s) [noise=true , address=77.90.31.18:4817 , piece_id=LAFPCQDUFJCZQQRGIIIMGKT3RY527BYIN55QPS2KGF4ADAWLI3UA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:36.436969825 (60h42m46.159532865s) [noise=true , address=64.49.13.138:4488 , piece_id=TE7I4CJMA767HHH5YKWMLJOE5LS5M32NGI76X62WQBO2G5NHGM4Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:36.569996585 (60h42m46.292559615s) [noise=true , address=77.90.31.16:4751 , piece_id=LJB7N2VOHYA43XZP32I4RPHRI4G63IWF33JVFBIQY56YFYIEAU2A, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:37.028571727 (60h42m46.751134767s) [noise=true , address=79.127.219.38:4001 , piece_id=KMTLQ2CLCS6ZF66SF7JSPUPPIWZVQFXYPCKDS3KIDDMPVJFLDUEA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:46.549073710 (60h42m56.27163675s) [noise=true , address=77.90.31.11:4576 , piece_id=7ZW7TOSVB5BPFZT4HGNCBBRCTESX5GNM5V5ULBLYU4MYUFAVZEJQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:52.538901001 (60h43m2.261464041s) [noise=true , address=77.90.31.12:4611 , piece_id=JTFTDCB2KJ3WJ75XDXUYGEN55QL4WKCKPG2TW5TE7FGT3TLQFO6A, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:18:52.716311530 (60h43m2.43887457s) [noise=true , address=45.91.116.9:4277 , piece_id=KO52SDQPK4MCQD7XAF7Q5ZCIKAGBVRGFPWKXZMW45C4TDWDIM56Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: <nil> Jul 27 00:19:33.377643345 (60h43m43.100206385s) [noise=true , address=77.90.31.17:4778 , piece_id=QIYIIBCZGQPNNOVHC7ZNF76VAACNJGMXTCO5EQUJZE4DNELQH6IQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: write tcp 188.166.225.170:45538->77.90.31.17:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377710294 (60h43m43.100273344s) [noise=true , address=104.36.82.53:4147 , piece_id=COCCHCNFRMA425SNFZLKTZNLCXYEEATE3IWP4KERGKJ72VL47BNA, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377770599 (60h43m43.100333639s) [noise=true , address=104.36.82.52:4119 , piece_id=27ZPC7G2IMJJ6GQWD3WZGE7RFZKPXKOSPUE7HO2SQHDW3KIBSR5Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: write tcp 188.166.225.170:48056->104.36.82.52:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377842056 (60h43m43.100405096s) [noise=true , address=104.36.82.46:4501 , piece_id=GJYUQA7AFKYED4N7H2ONXWTD4SQ7OLQTTVVHQDRQ2OV7HPP37QWQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: rpc: dial tcp 104.36.82.46:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377904786 (60h43m43.100467826s) [noise=true , address=104.36.82.55:4194 , piece_id=HNEE5JZ6BZ5JHGROQSSKDSBIGUMRLGDVLJJ6GSYA4KKIMBRVLKQQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 188.166.225.170:37680->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377910507 (60h43m43.100473537s) [noise=true , address=104.36.82.54:4190 , piece_id=SPKGL26ZFVI6VWJQIGBOG3BZZ2Z4RTPHUGLDQUIKZRWTEFRG463Q, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: write tcp 188.166.225.170:52512->104.36.82.54:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377921829 (60h43m43.100484869s) [noise=true , address=104.36.82.47:4546 , piece_id=44H6TXDATOOSKBWLNRBW6D77CUT4RBDONMXI34UOEAUGLQCBBJTQ, seg_pos={0 0}, upload=58226]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:33.377934783 (60h43m43.100497823s) [seg_pos={0 0}, upload=58226]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:19:33.610061660 (60h43m43.332624701s) [upload=58226]: Done uploading segments... Jul 27 00:19:33.610086097 (60h43m43.332649137s) [upload=58226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.234363198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.X] duration: 30.241082604s FAIL (2025-07-27 00:19:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.241107782s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.866204039s FAIL (2025-07-27 00:19:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.866227585s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X....] duration: 7.593078113s FAIL (2025-07-27 00:19:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.593117839s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XXX..X.....X......X...XX..XX.X.XX...X...............XX......] duration: 21.455656483s FAIL (2025-07-27 00:18:57 UTC) orbiter warning: context deadline exceeded Duration: 21.455667874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX] duration: 30.227517853s FAIL (2025-07-27 00:18:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== e of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:18:24.334097790 (60h42m34.05666082s) [upload=58224]: Starting upload Jul 27 00:18:24.334154699 (60h42m34.056717779s) [upload=58224]: Done starting upload Jul 27 00:18:24.334237908 (60h42m34.056800938s) [upload=58224]: Uploading segments... Jul 27 00:18:24.334437640 (60h42m34.05700067s) [upload=58224]: Got next segment. Inline: false Jul 27 00:18:24.334483638 (60h42m34.057046679s) [upload=58224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:24.334488578 (60h42m34.057051618s) [upload=58224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:24.564881187 (60h42m34.287444227s) [seg_pos={0 0}, upload=58224]: Begin upload segment... Jul 27 00:18:24.564929989 (60h42m34.28749303s) [seg_pos={0 0}, upload=58224]: Done begin upload segment. Jul 27 00:18:24.564954016 (60h42m34.287517056s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.564998090 (60h42m34.28756112s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565311169 (60h42m34.287874209s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565454935 (60h42m34.288017965s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565559765 (60h42m34.288122805s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565729580 (60h42m34.28829262s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.565840673 (60h42m34.288403713s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566026167 (60h42m34.288589207s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566132010 (60h42m34.28869505s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566382689 (60h42m34.288945719s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566620755 (60h42m34.289183795s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.566845164 (60h42m34.289408204s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567029938 (60h42m34.289592978s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567212567 (60h42m34.289775607s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567393123 (60h42m34.289956163s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:24.567580712 (60h42m34.290143742s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Uploading piece... Jul 27 00:18:26.259818721 (60h42m35.982381751s) [upload=58224]: Next returned nil segment Jul 27 00:18:26.259835624 (60h42m35.982398654s) [upload=58224]: Waiting for error group managing segments... Jul 27 00:18:26.584579627 (60h42m36.307142667s) [noise=true , address=45.91.116.4:4940 , piece_id=7HF5C3H6DWCKOQB3CUV52SAABJE65DQJNVQLCIQXO24DN4YNQ2XA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592038658 (60h42m36.314601688s) [noise=true , address=45.91.116.11:4118 , piece_id=5FFG373E2WJG5VD2S6LKVZXKN4LIWGXPCNV4JZ6LP44IHFWIG4QQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.592821997 (60h42m36.315385038s) [noise=true , address=45.91.116.9:4262 , piece_id=7OCDAJ2QHBLYCDF6ZT5SA4WECXTB7EBGIFCSELIHOPKW4PGLSQMA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.662347510 (60h42m36.38491055s) [noise=true , address=45.91.116.12:4225 , piece_id=U46ITCPS7W2Q6KQIFRZXNE5M6MKNPUDCXQLL3OAVTZHT6HSUKKSQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.679168664 (60h42m36.401731704s) [noise=true , address=45.91.116.8:4045 , piece_id=2HZUZVN7GUT3TAOOZKOE24VKB7QRQ6PUORSJHY4JSLRDBIETYEZA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.700554562 (60h42m36.423117592s) [noise=true , address=77.90.31.15:4698 , piece_id=UAM5RJB7KA5JQWYF7OHIITPR4Z7EMRW3CGOAWMPZET4P2JUA4SHQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:26.736485559 (60h42m36.459048589s) [noise=true , address=45.91.116.6:4973 , piece_id=OSU2YNUJLTDHCF6I2POZQQ7FWQJAL7YAR2J5T4WLGY2TTOJZPROA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:27.128265201 (60h42m36.850828241s) [noise=true , address=45.91.116.14:4190 , piece_id=CUBBIX5KWOK5B7IMDGUORGGWJTS65QDDQC2IGPXASPAPGILT4F2Q, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.113985780 (60h42m37.83654882s) [noise=true , address=77.90.31.12:4589 , piece_id=TNACB2TJZDENYZCDIMKLOHETTXWOVD2PPW4ULJVT23HETS73W3RQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:28.206936645 (60h42m37.929499685s) [noise=true , address=77.90.31.19:4843 , piece_id=ROHKR4ZZKKQ6IJKAPQZZRCGVLK7EGWHT3WLPNMJMQLNHMRKIAB4A, seg_pos={0 0}, upload=58224]: Done uploading piece. err: <nil> Jul 27 00:18:53.685476775 (60h43m3.408039815s) [noise=true , address=104.36.82.15:4258 , piece_id=EGDBHJ7JABUKWNELV6WCWCATPUC2XMKES2XZJDWFHFIB7Q4B26OQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685723547 (60h43m3.408286577s) [noise=true , address=104.36.82.3:4765 , piece_id=MWN6BJEY7NS76U5CAX3SYKJSBQ2WWZX2AZYHVAGDSUQGO4V4RRLA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:59780->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685745219 (60h43m3.408308249s) [noise=true , address=104.36.82.14:4222 , piece_id=SJYHZJT4ZIJPMSSKKQBP44HH66DJ3P7ADTF6G3ZSSRYQDIH7TRZQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685781728 (60h43m3.408344768s) [noise=true , address=104.36.82.9:4005 , piece_id=4TZEAYYOM4VSULBFN3LZWQNE6CJP5HP2DG3C366PVJ7DWLQGNQQQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685849078 (60h43m3.408412118s) [noise=true , address=104.36.82.12:4151 , piece_id=73DGK7UHINOZSKEUGGK3HNN2UDLBGGMUS76BVLAHDTGM4UPOVIMQ, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685877802 (60h43m3.408440842s) [noise=true , address=104.36.82.9:4007 , piece_id=V3XR2OGUMJT2AXOOYRERKNDRKKL6A2TZJ5QSTTIOU3Z2OU73J3HA, seg_pos={0 0}, upload=58224]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:53.685897851 (60h43m3.408460891s) [seg_pos={0 0}, upload=58224]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:53.911235189 (60h43m3.633798229s) [upload=58224]: Done uploading segments... Jul 27 00:18:53.911264825 (60h43m3.633827865s) [upload=58224]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.227539935s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.XXX..X.....X......X...XX..XX.X.XX...X...............XX.....] duration: 46.165234705s FAIL (2025-07-27 00:18:36 UTC) orbiter warning: context deadline exceeded Duration: 46.165259161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23834285s FAIL (2025-07-27 00:18:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 08]: Done starting upload Jul 27 00:17:33.372160085 (60h41m43.094723125s) [upload=58208]: Uploading segments... Jul 27 00:17:33.372280545 (60h41m43.094843585s) [upload=58208]: Got next segment. Inline: false Jul 27 00:17:33.372328317 (60h41m43.094891357s) [upload=58208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:17:33.372332986 (60h41m43.094896026s) [upload=58208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:17:33.875900842 (60h41m43.598463882s) [seg_pos={0 0}, upload=58208]: Begin upload segment... Jul 27 00:17:33.875947512 (60h41m43.598510552s) [seg_pos={0 0}, upload=58208]: Done begin upload segment. Jul 27 00:17:33.876010863 (60h41m43.598573903s) [noise=true , address=104.36.82.50:4031 , piece_id=R2RJEBPSLLYRLRQB2WHCHOVBNK24YXUBIGZS7EOPBFMPHMXOHGHQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.876070787 (60h41m43.598633827s) [noise=true , address=79.127.201.215:4352 , piece_id=HSFVXI5KKBKJ7NCQLDF7YTSPX2EEECKF6V4QVOCWYLWB7SX457EQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.876361574 (60h41m43.598924614s) [noise=true , address=79.127.205.234:4616 , piece_id=7U3HHSNVAGR76XF5F2YGWXLUBHYSKG3KMXNKICUIRBA3TELOOOYQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.876572206 (60h41m43.599135246s) [noise=true , address=104.36.82.46:4495 , piece_id=CNWHW4MLQQGKCLJCLQBN57WEGKZUBSHH4RQVBG6P2GK25RKMB7JA, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.876725049 (60h41m43.599288089s) [noise=true , address=104.36.82.47:4521 , piece_id=5XNYJU4BXJT32IHXYWGGYWWUWHBVSYUPETJAPDRQHRK5XAS7TDGA, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.876881338 (60h41m43.599444379s) [noise=true , address=79.127.205.245:4241 , piece_id=ZUONRL3GM4XVIXAIA5TMX24WTJIV53M72SLUPOPFTGX4BFP6YVOQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.876994735 (60h41m43.599557775s) [noise=true , address=77.90.31.13:4653 , piece_id=MJC5QP65ONGVZ2UVACAGOVPXOSTIHMVCPTSAS2TPSB3T7ZXF673A, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877190480 (60h41m43.59975352s) [noise=true , address=45.91.116.4:4964 , piece_id=UNVK4BDU5MHDB6JSLKDTWS6A3RBQAWWZYDOEQEV2VA4TAXN7FIHQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877220667 (60h41m43.599783707s) [noise=true , address=79.127.219.48:4074 , piece_id=UBCM5CZL3VKAZ6LMAHTZQJAI36Y4VDV4KZTTW3VE7GF24M23WSNQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877449084 (60h41m43.600012124s) [noise=true , address=79.127.205.246:4272 , piece_id=GM4HMAZJG6AJK5U6LVO3KELRRY7ZZBG3FLAT4WEWMZP4RNW37BLQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877508928 (60h41m43.600071958s) [noise=true , address=64.49.13.140:4508 , piece_id=LHG52Q7FNDME4RWUR3KHHD5LBK5FHNY2NONLDBYRAZ723QRJZRKQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877667712 (60h41m43.600230752s) [noise=true , address=79.127.220.101:4041 , piece_id=T3JZ4XQIZGZZBE4Z5VHOXLTKG6RVG33FW7LNYSNX3DULJCX5SCBQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877871954 (60h41m43.600434984s) [noise=true , address=45.91.116.9:4277 , piece_id=7OM5FD6LXQXI7XJAXGCWDV4FHBLGHF6RW2UJNJENKABIXER2YP6A, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.877920576 (60h41m43.600483616s) [noise=true , address=104.36.82.54:4186 , piece_id=BQJYVKPRKTR7GUX53YFPYEO4QYXAHIC2Q6OMOYRMOPQUWHQYJAPQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.878126390 (60h41m43.60068943s) [noise=true , address=79.127.205.235:4675 , piece_id=GJQW2YOFHLGLNHSARFBGRUGCA65MIWEZIYL7WH6YFO5LOZK5BRYQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.878131310 (60h41m43.60069435s) [noise=true , address=79.127.219.38:4004 , piece_id=OYYUKP3PTYJQS4LMFX27P2CL3C5VTCDTYVEZJUL6I7IKT2Y2BMQQ, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.878324780 (60h41m43.60088782s) [noise=true , address=77.90.31.11:4576 , piece_id=EMY4FKCAMITYCYKAPS62E7KR5AWKDLEQWDPKAHETRS6S6E3MCE3A, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:33.878519703 (60h41m43.601082743s) [noise=true , address=104.36.82.56:4236 , piece_id=HGTLRN55L5AVLOPG5FS3DTCGUN4FMBDW3XGUMUJVYXVPKRGVD24Q, seg_pos={0 0}, upload=58208]: Uploading piece... Jul 27 00:17:34.981405976 (60h41m44.703969016s) [upload=58208]: Next returned nil segment Jul 27 00:17:34.981424491 (60h41m44.703987531s) [upload=58208]: Waiting for error group managing segments... Jul 27 00:17:35.278460150 (60h41m45.00102319s) [noise=true , address=79.127.205.234:4616 , piece_id=7U3HHSNVAGR76XF5F2YGWXLUBHYSKG3KMXNKICUIRBA3TELOOOYQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:35.282160720 (60h41m45.00472376s) [noise=true , address=79.127.205.245:4241 , piece_id=ZUONRL3GM4XVIXAIA5TMX24WTJIV53M72SLUPOPFTGX4BFP6YVOQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:35.317465689 (60h41m45.040028729s) [noise=true , address=79.127.205.235:4675 , piece_id=GJQW2YOFHLGLNHSARFBGRUGCA65MIWEZIYL7WH6YFO5LOZK5BRYQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:35.535822701 (60h41m45.258385742s) [noise=true , address=79.127.205.246:4272 , piece_id=GM4HMAZJG6AJK5U6LVO3KELRRY7ZZBG3FLAT4WEWMZP4RNW37BLQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:35.762793635 (60h41m45.485356675s) [noise=true , address=79.127.219.38:4004 , piece_id=OYYUKP3PTYJQS4LMFX27P2CL3C5VTCDTYVEZJUL6I7IKT2Y2BMQQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:36.084062108 (60h41m45.806625148s) [noise=true , address=77.90.31.11:4576 , piece_id=EMY4FKCAMITYCYKAPS62E7KR5AWKDLEQWDPKAHETRS6S6E3MCE3A, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:36.608883379 (60h41m46.331446409s) [noise=true , address=77.90.31.13:4653 , piece_id=MJC5QP65ONGVZ2UVACAGOVPXOSTIHMVCPTSAS2TPSB3T7ZXF673A, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:39.041405645 (60h41m48.763968685s) [noise=true , address=79.127.219.48:4074 , piece_id=UBCM5CZL3VKAZ6LMAHTZQJAI36Y4VDV4KZTTW3VE7GF24M23WSNQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:39.079412553 (60h41m48.801975593s) [noise=true , address=79.127.220.101:4041 , piece_id=T3JZ4XQIZGZZBE4Z5VHOXLTKG6RVG33FW7LNYSNX3DULJCX5SCBQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:40.757101656 (60h41m50.479664697s) [noise=true , address=45.91.116.9:4277 , piece_id=7OM5FD6LXQXI7XJAXGCWDV4FHBLGHF6RW2UJNJENKABIXER2YP6A, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:43.495160447 (60h41m53.217723487s) [noise=true , address=64.49.13.140:4508 , piece_id=LHG52Q7FNDME4RWUR3KHHD5LBK5FHNY2NONLDBYRAZ723QRJZRKQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:17:47.080745559 (60h41m56.803308599s) [noise=true , address=45.91.116.4:4964 , piece_id=UNVK4BDU5MHDB6JSLKDTWS6A3RBQAWWZYDOEQEV2VA4TAXN7FIHQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:18:12.957752967 (60h42m22.680315997s) [noise=true , address=79.127.201.215:4352 , piece_id=HSFVXI5KKBKJ7NCQLDF7YTSPX2EEECKF6V4QVOCWYLWB7SX457EQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: <nil> Jul 27 00:18:32.732311750 (60h42m42.45487479s) [noise=true , address=104.36.82.47:4521 , piece_id=5XNYJU4BXJT32IHXYWGGYWWUWHBVSYUPETJAPDRQHRK5XAS7TDGA, seg_pos={0 0}, upload=58208]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: write tcp 188.166.225.170:51770->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:32.732362567 (60h42m42.454925608s) [noise=true , address=104.36.82.50:4031 , piece_id=R2RJEBPSLLYRLRQB2WHCHOVBNK24YXUBIGZS7EOPBFMPHMXOHGHQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 188.166.225.170:50678->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:32.732398025 (60h42m42.454961055s) [noise=true , address=104.36.82.56:4236 , piece_id=HGTLRN55L5AVLOPG5FS3DTCGUN4FMBDW3XGUMUJVYXVPKRGVD24Q, seg_pos={0 0}, upload=58208]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: write tcp 188.166.225.170:59522->104.36.82.56:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:32.732469632 (60h42m42.455032672s) [noise=true , address=104.36.82.46:4495 , piece_id=CNWHW4MLQQGKCLJCLQBN57WEGKZUBSHH4RQVBG6P2GK25RKMB7JA, seg_pos={0 0}, upload=58208]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:18:32.732525058 (60h42m42.455088098s) [noise=true , address=104.36.82.54:4186 , piece_id=BQJYVKPRKTR7GUX53YFPYEO4QYXAHIC2Q6OMOYRMOPQUWHQYJAPQ, seg_pos={0 0}, upload=58208]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: rpc: dial tcp 104.36.82.54:4186: i/o timeout; context deadline exceeded Jul 27 00:18:32.732582318 (60h42m42.455145358s) [seg_pos={0 0}, upload=58208]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:18:32.968878835 (60h42m42.691441875s) [upload=58208]: Done uploading segments... Jul 27 00:18:32.968906648 (60h42m42.691469678s) [upload=58208]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.238355835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.] duration: 7.035735839s FAIL (2025-07-27 00:18:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.035760406s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.849941744s FAIL (2025-07-27 00:18:11 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.849970518s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X...] duration: 6.538882597s FAIL (2025-07-27 00:18:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.538908527s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....X] duration: 30.235111243s FAIL (2025-07-27 00:18:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 038 (60h40m54.227618078s) [upload=58197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:16:44.795161809 (60h40m54.517724839s) [upload=58197]: Done uploading segments... Jul 27 00:16:44.795179844 (60h40m54.517742884s) [upload=58197]: Upload finished. err: <nil> Jul 27 00:17:33.962231008 (60h41m43.684794048s) [upload=58209]: Starting upload Jul 27 00:17:33.962310229 (60h41m43.684873269s) [upload=58209]: Done starting upload Jul 27 00:17:33.962390943 (60h41m43.684953983s) [upload=58209]: Uploading segments... Jul 27 00:17:33.962576528 (60h41m43.685139558s) [upload=58209]: Got next segment. Inline: false Jul 27 00:17:33.962616475 (60h41m43.685179515s) [upload=58209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:17:33.962621615 (60h41m43.685184655s) [upload=58209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:17:34.194429976 (60h41m43.916993016s) [seg_pos={0 0}, upload=58209]: Begin upload segment... Jul 27 00:17:34.194474481 (60h41m43.917037521s) [seg_pos={0 0}, upload=58209]: Done begin upload segment. Jul 27 00:17:34.194500090 (60h41m43.91706314s) [noise=true , address=77.90.31.10:4520 , piece_id=26JWCEJCLZYB73XLT2XIO5ONDV6MGCKAUD55ENT7D5YJQ4GCQUEA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.194504578 (60h41m43.917067619s) [noise=true , address=104.36.82.2:4731 , piece_id=DT7ALGELQZCFIP6Z7U7VOE5G5623ULD6PSVI7LFEQEWU2T27V5EQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.194886700 (60h41m43.91744974s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZFP2I5NAHKUVWZW7EYJOOYF32IO3T2R5CE2ZYNEYUEIXRCOOA4A, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.194934141 (60h41m43.917497181s) [noise=true , address=77.90.31.17:4771 , piece_id=QTMLXEIMKZ74EYLQVQIV4WNTQHO2RNSYZLJ5QUUZWD446ROSQ5WA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195273580 (60h41m43.91783662s) [noise=true , address=45.91.116.14:4189 , piece_id=GQKKQL2B52K2CNLYOLAXUDWZO73VFEGONGAEOEDCQMTT3BPIWZ3Q, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195591829 (60h41m43.918154879s) [noise=true , address=77.90.31.16:4734 , piece_id=V24CW3MXIP73N7IWYESHDAV4QXRZKYSZ2U4D3FQELK4O4XEBD5HA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195625543 (60h41m43.918188583s) [noise=true , address=77.90.31.13:4626 , piece_id=MIYCINRM67BQB3KRIAAF2WUN4B24BZKWAW4W2G6YPTRVQKYE4O7Q, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195886343 (60h41m43.918449623s) [noise=true , address=77.90.31.10:4517 , piece_id=LTIBGQWRDGTW2LPMFZNDTBP236TLVTOZGA4TMCJQL6LLJSRAXYTQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.196281128 (60h41m43.918844168s) [noise=true , address=77.90.31.14:4663 , piece_id=JGRSTHDJFOA3TFWRLA62DCX4JJJTJASM6NFN663UK2CA22Z7YGSA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197037776 (60h41m43.919600816s) [noise=true , address=45.91.116.5:4299 , piece_id=BQM6HZK2CNCV23SEXCBDJGYCICKXOAIXAVDBSLGYI56ZE7KKCDIA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197257166 (60h41m43.919820206s) [noise=true , address=45.91.116.11:4118 , piece_id=H2UNNR2QQ4ZFRLCC3D4Z4WSYC4IYHBBVQYG6FIMSIQV4XY2QL4TQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197455756 (60h41m43.920018796s) [noise=true , address=45.91.116.11:4119 , piece_id=VSLZZ77LFNQQKJJWFIHAGMTAJUNITHT3RK3DE7P6SJXQLDMSS4FQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197653474 (60h41m43.920216514s) [noise=true , address=45.91.116.10:4081 , piece_id=37OMV255QGHBEO7FEHMRL44LQYY4PKNNQVYIRVWWQJPN52P5CYIA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197888202 (60h41m43.920451242s) [noise=true , address=45.91.116.12:4227 , piece_id=WO6HD4FZWCTXBMM2LQC7GVNTZVNAPD4RJ64L5WIAFWNBIZU245MA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.198098745 (60h41m43.920661786s) [noise=true , address=104.36.82.8:4970 , piece_id=DBNR2K5WQVQIIT5XDMSXFDU5YBH3PUVWZKLA5M3I2HEFLACDM65A, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.198294289 (60h41m43.92085733s) [noise=true , address=104.36.82.8:4971 , piece_id=MZ3CIGVDC7LVVZNTNROF6OEQ5K7NTCCR5GT7NPLUOIYMPNZFZ5UA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:35.949021956 (60h41m45.671584996s) [upload=58209]: Next returned nil segment Jul 27 00:17:35.949041252 (60h41m45.671604292s) [upload=58209]: Waiting for error group managing segments... Jul 27 00:17:36.226962519 (60h41m45.949525559s) [noise=true , address=77.90.31.16:4734 , piece_id=V24CW3MXIP73N7IWYESHDAV4QXRZKYSZ2U4D3FQELK4O4XEBD5HA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.296215582 (60h41m46.018778622s) [noise=true , address=77.90.31.14:4663 , piece_id=JGRSTHDJFOA3TFWRLA62DCX4JJJTJASM6NFN663UK2CA22Z7YGSA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.305929036 (60h41m46.028492076s) [noise=true , address=45.91.116.11:4118 , piece_id=H2UNNR2QQ4ZFRLCC3D4Z4WSYC4IYHBBVQYG6FIMSIQV4XY2QL4TQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.311637579 (60h41m46.034200619s) [noise=true , address=45.91.116.10:4081 , piece_id=37OMV255QGHBEO7FEHMRL44LQYY4PKNNQVYIRVWWQJPN52P5CYIA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.361868666 (60h41m46.084431706s) [noise=true , address=45.91.116.5:4299 , piece_id=BQM6HZK2CNCV23SEXCBDJGYCICKXOAIXAVDBSLGYI56ZE7KKCDIA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.366642170 (60h41m46.0892052s) [noise=true , address=45.91.116.11:4119 , piece_id=VSLZZ77LFNQQKJJWFIHAGMTAJUNITHT3RK3DE7P6SJXQLDMSS4FQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.428801869 (60h41m46.151364909s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZFP2I5NAHKUVWZW7EYJOOYF32IO3T2R5CE2ZYNEYUEIXRCOOA4A, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.429246590 (60h41m46.15180963s) [noise=true , address=45.91.116.14:4189 , piece_id=GQKKQL2B52K2CNLYOLAXUDWZO73VFEGONGAEOEDCQMTT3BPIWZ3Q, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.462050493 (60h41m46.184613533s) [noise=true , address=45.91.116.12:4227 , piece_id=WO6HD4FZWCTXBMM2LQC7GVNTZVNAPD4RJ64L5WIAFWNBIZU245MA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:37.736988755 (60h41m47.459551795s) [noise=true , address=77.90.31.13:4626 , piece_id=MIYCINRM67BQB3KRIAAF2WUN4B24BZKWAW4W2G6YPTRVQKYE4O7Q, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:18:03.315118280 (60h42m13.03768132s) [noise=true , address=104.36.82.8:4970 , piece_id=DBNR2K5WQVQIIT5XDMSXFDU5YBH3PUVWZKLA5M3I2HEFLACDM65A, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:44300->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315160911 (60h42m13.037723951s) [noise=true , address=104.36.82.8:4971 , piece_id=MZ3CIGVDC7LVVZNTNROF6OEQ5K7NTCCR5GT7NPLUOIYMPNZFZ5UA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:57224->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315259931 (60h42m13.037822971s) [noise=true , address=77.90.31.17:4771 , piece_id=QTMLXEIMKZ74EYLQVQIV4WNTQHO2RNSYZLJ5QUUZWD446ROSQ5WA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:55870->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315411350 (60h42m13.03797439s) [noise=true , address=77.90.31.10:4520 , piece_id=26JWCEJCLZYB73XLT2XIO5ONDV6MGCKAUD55ENT7D5YJQ4GCQUEA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:53376->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315444585 (60h42m13.038007625s) [noise=true , address=104.36.82.2:4731 , piece_id=DT7ALGELQZCFIP6Z7U7VOE5G5623ULD6PSVI7LFEQEWU2T27V5EQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:36034->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315573330 (60h42m13.038136371s) [noise=true , address=77.90.31.10:4517 , piece_id=LTIBGQWRDGTW2LPMFZNDTBP236TLVTOZGA4TMCJQL6LLJSRAXYTQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:38714->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.235134668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X] duration: 30.260495401s FAIL (2025-07-27 00:17:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.260508746s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.851260248s FAIL (2025-07-27 00:17:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.851295877s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239180401s FAIL (2025-07-27 00:17:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ABL2LSY3XMCZNUXCSMFO7TZZLQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.326910813 (60h40m43.049473854s) [noise=true , address=104.36.82.52:4098 , piece_id=3V5VPN5QFWTKSMYEBONIQVBFXDPONMIPV7AW5ADMZ5TFSDHYCZFA, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327042626 (60h40m43.049605666s) [noise=true , address=77.90.31.15:4731 , piece_id=3BXDCVJYGS6AFH2PRXS6QXN72PX2OJHUPNCVA33MK5SOEMZZPIFA, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327308224 (60h40m43.049871264s) [noise=true , address=104.36.82.47:4546 , piece_id=ZMYG6R6HYPQNWFQXN57WU6RK3IJRUPG5VNU7XD4JPVFUJTOYU4UQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327362669 (60h40m43.049925699s) [noise=true , address=79.127.205.246:4275 , piece_id=X5EJR5TKWOTWRMDTJDRXB5B6HGVAPNF3E43OIPEQ2NQBZTTCMEHA, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327620542 (60h40m43.050183582s) [noise=true , address=104.36.82.53:4141 , piece_id=TKMODPYCU2JN5CWQMLMJVTHTFKIZGRODE3BYDDTINQPTCFV2HCDQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327637194 (60h40m43.050200224s) [noise=true , address=79.127.205.234:4620 , piece_id=RQKRPA4AG7IQ5FHBHOD73VFBVRCPT5ERGVTJMTC6LAG65Z46HUIQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327868075 (60h40m43.050431115s) [noise=true , address=79.127.220.101:4048 , piece_id=YDZEOSWHU5VKEQUX4KWVEBOBWQDPQD6EHLPOSPP5FPBM5E6TE34Q, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327887683 (60h40m43.050450723s) [noise=true , address=77.90.31.21:4908 , piece_id=7POCZM436STIDLGT3CUAUTUQMMREXJWRHGQH6OZQBLTZASLRUUXQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328181115 (60h40m43.050744155s) [noise=true , address=104.36.82.12:4181 , piece_id=FIKOB2EPEKOOMTDFPSYQWW5KYAXVJO7COD6BINI2EZCKI27QFJQQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328215540 (60h40m43.05077857s) [noise=true , address=104.36.82.21:4507 , piece_id=QEMOQHL74KEZ2VCANXRT2LXYZVIZR2EKY2VEJJQLKJHRUQ3C653A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328433457 (60h40m43.050996497s) [noise=true , address=79.127.205.245:4241 , piece_id=74QK5TUK2GVVFFVTEXLP5DMF7ZF554VWE3PJIKGZHTKHEJZFZFYQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328632177 (60h40m43.051195217s) [noise=true , address=104.36.82.55:4199 , piece_id=QZ4MNNJWW3SUSSLDQNQEHQ6CH2XN5GOPQWGLQ3CDVJ3KBECUBGQQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328686642 (60h40m43.051249672s) [noise=true , address=104.36.82.49:4990 , piece_id=TVMAWIQA3XXIPT26SRWADBNBFCEH457ORJVLAAA34XVA36OXI27A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.329217747 (60h40m43.051780828s) [noise=true , address=77.90.31.10:4545 , piece_id=KAIY7BGG2KLVBUEZ2DVBBVJSVHFIOJZDOGAR7RBE52QNAOZD5C2A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.329740387 (60h40m43.052303447s) [noise=true , address=79.127.201.214:4321 , piece_id=LCS4JM76YANYBADJYD4SHZCTGGA3Z5YHTGRW5JYM5EAAALK2NZ6A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:34.428894880 (60h40m44.15145792s) [upload=58191]: Next returned nil segment Jul 27 00:16:34.428915189 (60h40m44.151478229s) [upload=58191]: Waiting for error group managing segments... Jul 27 00:16:34.733722888 (60h40m44.456285928s) [noise=true , address=79.127.205.245:4241 , piece_id=74QK5TUK2GVVFFVTEXLP5DMF7ZF554VWE3PJIKGZHTKHEJZFZFYQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:34.734393011 (60h40m44.456956051s) [noise=true , address=79.127.205.234:4620 , piece_id=RQKRPA4AG7IQ5FHBHOD73VFBVRCPT5ERGVTJMTC6LAG65Z46HUIQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:34.737835798 (60h40m44.460398838s) [noise=true , address=79.127.205.233:4653 , piece_id=7E3XX3YHGIBK4Y5A37ELQREDTTK6ZAP34C45RE3B5BEKBGZEACOQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:34.790383105 (60h40m44.512946145s) [noise=true , address=79.127.205.246:4275 , piece_id=X5EJR5TKWOTWRMDTJDRXB5B6HGVAPNF3E43OIPEQ2NQBZTTCMEHA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:37.963538347 (60h40m47.686101387s) [noise=true , address=77.90.31.18:4832 , piece_id=MG4YIJP563M47RJLS32ESPSFKAABL2LSY3XMCZNUXCSMFO7TZZLQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:38.305732748 (60h40m48.028295788s) [noise=true , address=79.127.201.214:4321 , piece_id=LCS4JM76YANYBADJYD4SHZCTGGA3Z5YHTGRW5JYM5EAAALK2NZ6A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:38.726111399 (60h40m48.448674429s) [noise=true , address=79.127.220.101:4048 , piece_id=YDZEOSWHU5VKEQUX4KWVEBOBWQDPQD6EHLPOSPP5FPBM5E6TE34Q, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:50.140254691 (60h40m59.862817731s) [noise=true , address=77.90.31.15:4731 , piece_id=3BXDCVJYGS6AFH2PRXS6QXN72PX2OJHUPNCVA33MK5SOEMZZPIFA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:17:32.242806642 (60h41m41.965369683s) [noise=true , address=77.90.31.10:4545 , piece_id=KAIY7BGG2KLVBUEZ2DVBBVJSVHFIOJZDOGAR7RBE52QNAOZD5C2A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: write tcp 188.166.225.170:56242->77.90.31.10:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242864313 (60h41m41.965427353s) [noise=true , address=104.36.82.47:4546 , piece_id=ZMYG6R6HYPQNWFQXN57WU6RK3IJRUPG5VNU7XD4JPVFUJTOYU4UQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242896165 (60h41m41.965459205s) [noise=true , address=104.36.82.21:4507 , piece_id=QEMOQHL74KEZ2VCANXRT2LXYZVIZR2EKY2VEJJQLKJHRUQ3C653A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242937413 (60h41m41.965500453s) [noise=true , address=104.36.82.55:4199 , piece_id=QZ4MNNJWW3SUSSLDQNQEHQ6CH2XN5GOPQWGLQ3CDVJ3KBECUBGQQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242971909 (60h41m41.965534949s) [noise=true , address=77.90.31.21:4908 , piece_id=7POCZM436STIDLGT3CUAUTUQMMREXJWRHGQH6OZQBLTZASLRUUXQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: read tcp 188.166.225.170:34178->77.90.31.21:4908: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243018638 (60h41m41.965581668s) [noise=true , address=104.36.82.12:4181 , piece_id=FIKOB2EPEKOOMTDFPSYQWW5KYAXVJO7COD6BINI2EZCKI27QFJQQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: write tcp 188.166.225.170:52330->104.36.82.12:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243051611 (60h41m41.965614651s) [noise=true , address=104.36.82.52:4098 , piece_id=3V5VPN5QFWTKSMYEBONIQVBFXDPONMIPV7AW5ADMZ5TFSDHYCZFA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1SoFNdwfh4p5tXUkpfjQaev6tdA2ZdhsXUFvZNBsy5QrJ9zdqb, address:104.36.82.52:4098): protocol: write tcp 188.166.225.170:41640->104.36.82.52:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243095696 (60h41m41.965658736s) [noise=true , address=104.36.82.46:4509 , piece_id=TW4DDBQPHTSLLCULGYAMROCPWSKWNG5D57MA4L62W2XQ5GTQNBNA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: rpc: dial tcp 104.36.82.46:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243113570 (60h41m41.9656766s) [noise=true , address=104.36.82.53:4141 , piece_id=TKMODPYCU2JN5CWQMLMJVTHTFKIZGRODE3BYDDTINQPTCFV2HCDQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243422882 (60h41m41.965985932s) [noise=true , address=104.36.82.49:4990 , piece_id=TVMAWIQA3XXIPT26SRWADBNBFCEH457ORJVLAAA34XVA36OXI27A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1gDzCKcMaLTisHw68GYkMbcMJF7rbKnUt48etS8Q7KeiXedbU4, address:104.36.82.49:4990): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:17:32.243446277 (60h41m41.966009317s) [seg_pos={0 0}, upload=58191]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:17:32.480047178 (60h41m42.202610218s) [upload=58191]: Done uploading segments... Jul 27 00:17:32.480072977 (60h41m42.202636017s) [upload=58191]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.239192936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..XXX..X.....X......X...XX..XX.X.XX...X...............XX....] duration: 40.335758121s FAIL (2025-07-27 00:17:17 UTC) orbiter warning: context deadline exceeded Duration: 40.335797316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..] duration: 6.178601714s FAIL (2025-07-27 00:16:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.178623836s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)