Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234493291s FAIL (2025-07-27 00:35:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .947246070 (60h58m10.66980911s) [noise=true , address=79.127.205.234:4606 , piece_id=BPO2QLZLYDMLDFG75PZWRQEDLFVYHATXPOSFEHNOW5TTQVHJNCDQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.947411467 (60h58m10.669974507s) [noise=true , address=79.127.201.214:4321 , piece_id=QNIQHFLYGDYMJAKX34J25OWNOSBJ6ARDNOQN6ORKKJJ5YCPN5BIQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.947611719 (60h58m10.67017476s) [noise=true , address=104.36.82.56:4243 , piece_id=NV4M2T6H2JQ3BNLQEWTDIFWQSRASLNC6XAMUTTSROGZAOUBLH2FQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.947730897 (60h58m10.670293937s) [noise=true , address=104.36.82.55:4211 , piece_id=ZX3TNIL6OVFU7CL6HQWWBB2D2OZG3QQF2PCA4FF2CRD2TPD5PURQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948008538 (60h58m10.670571578s) [noise=true , address=79.127.201.215:4349 , piece_id=5JW3GKNWRETNYWAMWDTDCYG4STMUG5AJTJARZS2EDCMUETXWLUAQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948329142 (60h58m10.670892182s) [noise=true , address=104.36.82.47:4517 , piece_id=3SPHKVUJIB4LTR32O6P6DOHKD2LZZ3BZS42GDJAH6MML4P2JH4QA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948347508 (60h58m10.670910548s) [noise=true , address=104.36.82.4:4828 , piece_id=TGD6QOF3TQDNNFMS2O3V752GGMUFJYHVO63MINLF4MPFVSE6D46A, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948617704 (60h58m10.671180744s) [noise=true , address=45.91.116.10:4093 , piece_id=SBUAEHFRMRG5MF3K7CIT54MOGMINSBDMMAPMUCAN7MICW4HNZQHA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948722424 (60h58m10.671285464s) [noise=true , address=79.127.205.235:4688 , piece_id=YJHAWDUPMGXZLOXM443YGCBFAC6VC5PQTODFIPSGOSR34UEUXJVQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948856280 (60h58m10.671419321s) [noise=true , address=104.36.82.54:4165 , piece_id=RQUNZJHXQZX2QSWBPMFNGWSZGWGBT4PQHA2BOHHIKRSX4OKUQK6A, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949040964 (60h58m10.671603994s) [noise=true , address=79.127.219.38:4001 , piece_id=BK54RX4RCXMUMPTVO3SNEMCUDT3Q7VHPLEU2QPSYV2R4QJJAEMRA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949175291 (60h58m10.671738331s) [noise=true , address=104.36.82.49:4978 , piece_id=U5TL3JMP3WMY6WXQAIIXWW3NWTRMUUSYCEUM5EMJRK67PELHGSBQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949339656 (60h58m10.671902696s) [noise=true , address=156.146.43.225:4449 , piece_id=LVQQLBP5XGRQJLIWFJH6L2HJHALCI4KP5XQDURUJJUUUYVVZDFEA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949404449 (60h58m10.671967489s) [noise=true , address=77.90.31.10:4530 , piece_id=VXE6EFJ2FLEEDNJAFFQIN3RGM5UBJXR6YER7CEN65RDPUYF53CCQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949647995 (60h58m10.672211036s) [noise=true , address=79.127.205.246:4288 , piece_id=47LEBRSACX5JQM426GON2AE6TMOPGP2TMGQFEXFKU5NJONHSZMAA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.954106295 (60h58m10.676669335s) [noise=true , address=104.36.82.52:4085 , piece_id=7MLVFJ64SLANN6PHAS2BQQLCYJ3JNPIRFDXGJU4ONGPEVC3FPRMA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:02.075052567 (60h58m11.797615607s) [upload=58466]: Next returned nil segment Jul 27 00:34:02.075075701 (60h58m11.797638741s) [upload=58466]: Waiting for error group managing segments... Jul 27 00:34:02.374234097 (60h58m12.096797137s) [noise=true , address=79.127.205.234:4606 , piece_id=BPO2QLZLYDMLDFG75PZWRQEDLFVYHATXPOSFEHNOW5TTQVHJNCDQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.375144979 (60h58m12.097708029s) [noise=true , address=79.127.205.246:4288 , piece_id=47LEBRSACX5JQM426GON2AE6TMOPGP2TMGQFEXFKU5NJONHSZMAA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.386272419 (60h58m12.108835459s) [noise=true , address=79.127.205.235:4688 , piece_id=YJHAWDUPMGXZLOXM443YGCBFAC6VC5PQTODFIPSGOSR34UEUXJVQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.865938136 (60h58m12.588501177s) [noise=true , address=156.146.43.225:4449 , piece_id=LVQQLBP5XGRQJLIWFJH6L2HJHALCI4KP5XQDURUJJUUUYVVZDFEA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.867764060 (60h58m12.5903271s) [noise=true , address=79.127.219.38:4001 , piece_id=BK54RX4RCXMUMPTVO3SNEMCUDT3Q7VHPLEU2QPSYV2R4QJJAEMRA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.994582891 (60h58m12.717145921s) [noise=true , address=79.127.201.214:4321 , piece_id=QNIQHFLYGDYMJAKX34J25OWNOSBJ6ARDNOQN6ORKKJJ5YCPN5BIQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:03.382397019 (60h58m13.104960059s) [noise=true , address=64.49.13.26:4953 , piece_id=ZXNFKT5IJ5MIXNAKTOX3GL6PG3AROWLQWUSIYX42HF7XRJM2FR3A, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:34:04.630471883 (60h58m14.353034923s) [noise=true , address=45.91.116.10:4093 , piece_id=SBUAEHFRMRG5MF3K7CIT54MOGMINSBDMMAPMUCAN7MICW4HNZQHA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:34:04.787587931 (60h58m14.510150971s) [noise=true , address=77.90.31.10:4530 , piece_id=VXE6EFJ2FLEEDNJAFFQIN3RGM5UBJXR6YER7CEN65RDPUYF53CCQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:29.756831010 (60h58m39.47939405s) [noise=true , address=79.127.201.215:4349 , piece_id=5JW3GKNWRETNYWAMWDTDCYG4STMUG5AJTJARZS2EDCMUETXWLUAQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:59.866440719 (60h59m9.589003759s) [noise=true , address=104.36.82.21:4489 , piece_id=GK7F7VIIDKRHGNO4ZTKRRUYSRYXU7L5PSYKGYFPIYKCXOFLR76KA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: rpc: dial tcp 104.36.82.21:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866482488 (60h59m9.589045528s) [noise=true , address=104.36.82.55:4211 , piece_id=ZX3TNIL6OVFU7CL6HQWWBB2D2OZG3QQF2PCA4FF2CRD2TPD5PURQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: rpc: dial tcp 104.36.82.55:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866510051 (60h59m9.589073091s) [noise=true , address=104.36.82.47:4517 , piece_id=3SPHKVUJIB4LTR32O6P6DOHKD2LZZ3BZS42GDJAH6MML4P2JH4QA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866547573 (60h59m9.589110613s) [noise=true , address=104.36.82.56:4243 , piece_id=NV4M2T6H2JQ3BNLQEWTDIFWQSRASLNC6XAMUTTSROGZAOUBLH2FQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866564525 (60h59m9.589127565s) [noise=true , address=104.36.82.4:4828 , piece_id=TGD6QOF3TQDNNFMS2O3V752GGMUFJYHVO63MINLF4MPFVSE6D46A, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:42892->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866622546 (60h59m9.589185606s) [noise=true , address=104.36.82.54:4165 , piece_id=RQUNZJHXQZX2QSWBPMFNGWSZGWGBT4PQHA2BOHHIKRSX4OKUQK6A, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866627927 (60h59m9.589190967s) [noise=true , address=104.36.82.49:4978 , piece_id=U5TL3JMP3WMY6WXQAIIXWW3NWTRMUUSYCEUM5EMJRK67PELHGSBQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 188.166.225.170:53440->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866663966 (60h59m9.589226996s) [noise=true , address=104.36.82.52:4085 , piece_id=7MLVFJ64SLANN6PHAS2BQQLCYJ3JNPIRFDXGJU4ONGPEVC3FPRMA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:37302->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866688292 (60h59m9.589251332s) [seg_pos={0 0}, upload=58466]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:35:00.098519222 (60h59m9.821082262s) [upload=58466]: Done uploading segments... Jul 27 00:35:00.098551884 (60h59m9.821114924s) [upload=58466]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.234516887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................................X............] duration: 2.045918278s FAIL (2025-07-27 00:34:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.045954647s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX] duration: 30.236702967s FAIL (2025-07-27 00:34:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.236726912s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.X...XX..XX.X.XX...X...............XX......X...XXX..XXX..X..] duration: 37.218755825s FAIL (2025-07-27 00:34:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.21881052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.X] duration: 30.232284392s FAIL (2025-07-27 00:34:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 52]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:41402->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:23.530660034 (60h57m33.253223074s) [seg_pos={0 0}, upload=58452]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:33:23.530738865 (60h57m33.253301905s) [upload=58452]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:33:23.530743704 (60h57m33.253306744s) [upload=58452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:33:23.967037040 (60h57m33.68960008s) [upload=58452]: Done uploading segments... Jul 27 00:33:23.967054363 (60h57m33.689617403s) [upload=58452]: Upload finished. err: <nil> Jul 27 00:33:58.546683818 (60h58m8.269246848s) [upload=58464]: Starting upload Jul 27 00:33:58.546743301 (60h58m8.269306341s) [upload=58464]: Done starting upload Jul 27 00:33:58.546831620 (60h58m8.26939466s) [upload=58464]: Uploading segments... Jul 27 00:33:58.546919329 (60h58m8.269482369s) [upload=58464]: Got next segment. Inline: false Jul 27 00:33:58.546951891 (60h58m8.269514931s) [upload=58464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:33:58.546956349 (60h58m8.269519389s) [upload=58464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:33:58.864936927 (60h58m8.587499967s) [seg_pos={0 0}, upload=58464]: Begin upload segment... Jul 27 00:33:58.864976662 (60h58m8.587539703s) [seg_pos={0 0}, upload=58464]: Done begin upload segment. Jul 27 00:33:58.864998124 (60h58m8.587561164s) [noise=true , address=77.90.31.12:4589 , piece_id=PNADWDRY6H27UCL5QWEQIVTIOK4GFPA4STOY4IG5HTLQG6LITTXA, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.865062998 (60h58m8.587626028s) [noise=true , address=45.91.116.4:4938 , piece_id=JASSU2ETESAOGKKFP4W2E5E5LSH3CFIO6V63UMU6STEQANAFQLOA, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.865353744 (60h58m8.587916784s) [noise=true , address=77.90.31.18:4805 , piece_id=6JEDCAQDP6XTIFKDZHHNEVOKMZ5S3ZK4OSZG5SCE2TS66HCYCDKA, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.865482740 (60h58m8.58804577s) [noise=true , address=77.90.31.10:4518 , piece_id=SQY62GMK3DAKSENENJPLZRYHWXWNB64RNFDDIWOG6UN7RPYXXPAA, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.865620414 (60h58m8.588183454s) [noise=true , address=104.36.82.14:4222 , piece_id=O4KW7YDMWTNB3Z6U67FHTFRHWUXQK3BFERC2PYECKCW6SHEZBLRQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.865738561 (60h58m8.588301601s) [noise=true , address=45.91.116.10:4082 , piece_id=7U2R2YG747JHUT5I3BFDE4MX54PT2TLWPSJUKWXHENW6OCSAHW6A, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.865892835 (60h58m8.588455875s) [noise=true , address=104.36.82.7:4911 , piece_id=X25XWWIEVG5OH6CEQJDGQUXHNTFXG7YAYHPBLWDGIOZRZXWNDY5Q, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.866003286 (60h58m8.588566326s) [noise=true , address=104.36.82.6:4876 , piece_id=VQ3STHBAMIK37ZLIVZYAATT477WYLFM6IN3YXBMLDSQOM6BFD55A, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.866175797 (60h58m8.588738837s) [noise=true , address=45.91.116.11:4117 , piece_id=OGSDGXDPHUV3PFW65ENJHO6EYDGAPNWCI5BULNIY7G2TLWEFVLTQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.866264887 (60h58m8.588827927s) [noise=true , address=45.91.116.8:4048 , piece_id=VOONR7QZCM7Y4OS2P3ZSNJHQEG73BTUSRZNIFI3WMZAHSP4ZBJGQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.866472053 (60h58m8.589035093s) [noise=true , address=45.91.116.13:4154 , piece_id=T23CFIBV6OJYZQDIOQRFMYWFD4DFMDSFOBAKQODTS5QJSSLMNXUA, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.866491601 (60h58m8.589054641s) [noise=true , address=45.91.116.9:4263 , piece_id=S5FTUO42QXBQFVKOZI7WLPH3IYYZT46WRL5JHJXRUFY53YVBBVGQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.866690151 (60h58m8.589253191s) [noise=true , address=45.91.116.5:4299 , piece_id=PJM5LD4VWSWN6QOZQDI5VWDO47OOFMAJYXKK2Q6ZKD27X6ZAYYAQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.867207400 (60h58m8.58977043s) [noise=true , address=45.91.116.12:4225 , piece_id=INGEMSQ2DB7LWBWJOZVHJL6JVMZCQZ3L2ZJXFKZLNJSWCSWKXWYA, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.867930674 (60h58m8.590493714s) [noise=true , address=45.91.116.6:4975 , piece_id=UD7THW7SQCQ46B6KYUVPIHFISKNFUMEZAVEJCYJLABZEYCMVKZQQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:33:58.868625995 (60h58m8.591189035s) [noise=true , address=77.90.31.14:4663 , piece_id=DYFA3LKSFOO2KEP5WL2BQ7V7YMW67PJCVUFQJLNIR2N2CJ4TNGQQ, seg_pos={0 0}, upload=58464]: Uploading piece... Jul 27 00:34:00.627234106 (60h58m10.349797146s) [upload=58464]: Next returned nil segment Jul 27 00:34:00.627249855 (60h58m10.349812885s) [upload=58464]: Waiting for error group managing segments... Jul 27 00:34:00.905984314 (60h58m10.628547354s) [noise=true , address=45.91.116.9:4263 , piece_id=S5FTUO42QXBQFVKOZI7WLPH3IYYZT46WRL5JHJXRUFY53YVBBVGQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.917414654 (60h58m10.639977694s) [noise=true , address=45.91.116.10:4082 , piece_id=7U2R2YG747JHUT5I3BFDE4MX54PT2TLWPSJUKWXHENW6OCSAHW6A, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.961335657 (60h58m10.683898697s) [noise=true , address=45.91.116.8:4048 , piece_id=VOONR7QZCM7Y4OS2P3ZSNJHQEG73BTUSRZNIFI3WMZAHSP4ZBJGQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.969235071 (60h58m10.691798101s) [noise=true , address=45.91.116.12:4225 , piece_id=INGEMSQ2DB7LWBWJOZVHJL6JVMZCQZ3L2ZJXFKZLNJSWCSWKXWYA, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.982911458 (60h58m10.705474498s) [noise=true , address=45.91.116.6:4975 , piece_id=UD7THW7SQCQ46B6KYUVPIHFISKNFUMEZAVEJCYJLABZEYCMVKZQQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.985020112 (60h58m10.707583153s) [noise=true , address=77.90.31.14:4663 , piece_id=DYFA3LKSFOO2KEP5WL2BQ7V7YMW67PJCVUFQJLNIR2N2CJ4TNGQQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.986222413 (60h58m10.708785453s) [noise=true , address=45.91.116.5:4299 , piece_id=PJM5LD4VWSWN6QOZQDI5VWDO47OOFMAJYXKK2Q6ZKD27X6ZAYYAQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:00.993292360 (60h58m10.7158554s) [noise=true , address=45.91.116.13:4154 , piece_id=T23CFIBV6OJYZQDIOQRFMYWFD4DFMDSFOBAKQODTS5QJSSLMNXUA, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:01.033755256 (60h58m10.756318296s) [noise=true , address=45.91.116.4:4938 , piece_id=JASSU2ETESAOGKKFP4W2E5E5LSH3CFIO6V63UMU6STEQANAFQLOA, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:01.050360577 (60h58m10.772923617s) [noise=true , address=45.91.116.11:4117 , piece_id=OGSDGXDPHUV3PFW65ENJHO6EYDGAPNWCI5BULNIY7G2TLWEFVLTQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:03.185484500 (60h58m12.90804757s) [noise=true , address=77.90.31.12:4589 , piece_id=PNADWDRY6H27UCL5QWEQIVTIOK4GFPA4STOY4IG5HTLQG6LITTXA, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:05.967621472 (60h58m15.690184512s) [noise=true , address=77.90.31.10:4518 , piece_id=SQY62GMK3DAKSENENJPLZRYHWXWNB64RNFDDIWOG6UN7RPYXXPAA, seg_pos={0 0}, upload=58464]: Done uploading piece. err: <nil> Jul 27 00:34:27.903769905 (60h58m37.626332945s) [noise=true , address=104.36.82.14:4222 , piece_id=O4KW7YDMWTNB3Z6U67FHTFRHWUXQK3BFERC2PYECKCW6SHEZBLRQ, seg_pos={0 0}, upload=58464]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:56564->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:27.903825401 (60h58m37.626388441s) [noise=true , address=104.36.82.6:4876 , piece_id=VQ3STHBAMIK37ZLIVZYAATT477WYLFM6IN3YXBMLDSQOM6BFD55A, seg_pos={0 0}, upload=58464]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:27.903903661 (60h58m37.626466711s) [noise=true , address=104.36.82.7:4911 , piece_id=X25XWWIEVG5OH6CEQJDGQUXHNTFXG7YAYHPBLWDGIOZRZXWNDY5Q, seg_pos={0 0}, upload=58464]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:34868->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:27.903930753 (60h58m37.626493793s) [noise=true , address=77.90.31.18:4805 , piece_id=6JEDCAQDP6XTIFKDZHHNEVOKMZ5S3ZK4OSZG5SCE2TS66HCYCDKA, seg_pos={0 0}, upload=58464]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:58034->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:27.903947314 (60h58m37.626510354s) [seg_pos={0 0}, upload=58464]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:34:28.134454290 (60h58m37.85701738s) [upload=58464]: Done uploading segments... Jul 27 00:34:28.134480620 (60h58m37.85704366s) [upload=58464]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232304009s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..X...] duration: 27.043305243s FAIL (2025-07-27 00:34:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.043325052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................................X...........] duration: 1.965857879s FAIL (2025-07-27 00:33:59 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.965903125s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232411715s FAIL (2025-07-27 00:33:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== oad=58445]: Uploading piece... Jul 27 00:32:54.783008114 (60h57m4.505571154s) [noise=true , address=104.36.82.12:4181 , piece_id=6AAWFKHIIIZ5IYJNEY6B4T3POYNGKZKO5NBPA3ABGXPK7SAVJV2Q, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.783021119 (60h57m4.505584169s) [noise=true , address=104.36.82.52:4118 , piece_id=OMDRZGC7WKORZYQHCHGIMKCSNVGVQTRXRXQW4CMJDNXZLFBTDNYQ, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.783639833 (60h57m4.506202893s) [noise=true , address=79.127.201.215:4334 , piece_id=VOO7STLQ2BHHYH6S3VOBOV6PA7SRYK7WD6BM4NULH3WNRZ5YX57Q, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.784170549 (60h57m4.506733579s) [noise=true , address=79.127.220.101:4041 , piece_id=3JOMURZ4NGLVADZNPRJG6TNUYQN2RYOU55FVMKTXYNKKYUWTBS3A, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.784375420 (60h57m4.50693846s) [noise=true , address=79.127.205.246:4273 , piece_id=JDUVUIVGHDAKPHWG6OT5TTUXACYUOZNAUV7LBJUU6A3DXZ4EPFYQ, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.784387784 (60h57m4.506950825s) [noise=true , address=104.36.82.46:4499 , piece_id=UQNKWA6VJBN3HGENSJ5MQA3327YUEGJVNS2BQBZCTXZZLGACZ2RA, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.784721603 (60h57m4.507284643s) [noise=true , address=79.127.220.98:4422 , piece_id=U255QKEEUF7DTIRBJFF4KGEWTKLKG3SLF6MBWYPMMLUPO7EDALHA, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.785161415 (60h57m4.507724465s) [noise=true , address=104.36.82.56:4244 , piece_id=H7IS6EN5K52CPUUV6HMB5IF2O42NUAGGYGBFI25O2SW7ZWOHGNUA, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.785225697 (60h57m4.507788737s) [noise=true , address=104.36.82.47:4541 , piece_id=ASRUJWGUUJMPGCT3O34C6PXB2N67WVTBQX4RGJ3WDM5KFGOUQWOA, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.785454625 (60h57m4.508017665s) [noise=true , address=79.127.205.245:4224 , piece_id=6HOU6RPHEUROKJQOUY6ZQJHBSVHGISDQAULMQW74HFDXL6IOIN2Q, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.785663425 (60h57m4.508226465s) [noise=true , address=77.90.31.21:4908 , piece_id=XXHF5R3QG63IMUYGSKR6LGQ75L5MBIA6P2M4CBR5HSHM52VAT75Q, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.786004096 (60h57m4.508567136s) [noise=true , address=77.90.31.13:4653 , piece_id=5BB4YP75W5JS35CVXTFNGJI4KCVQTNLK3X754WTQYDM6UL3TUCJA, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.786230200 (60h57m4.50879324s) [noise=true , address=104.36.82.54:4184 , piece_id=WS3QVDC5JQ3HKPJZX6YKR3VRLECQEKF7I5WKZ47ALJE72DEQLKVQ, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:54.786374566 (60h57m4.508937596s) [noise=true , address=104.36.82.50:4031 , piece_id=67Y72CYO74MH6OR5SN4NYAGDFWWI2VD6LWO4RESH34IX7BVPN2WQ, seg_pos={0 0}, upload=58445]: Uploading piece... Jul 27 00:32:55.772744102 (60h57m5.495307143s) [noise=true , address=79.127.220.98:4422 , piece_id=U255QKEEUF7DTIRBJFF4KGEWTKLKG3SLF6MBWYPMMLUPO7EDALHA, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE, address:79.127.220.98:4422): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:55.879233627 (60h57m5.601796667s) [upload=58445]: Next returned nil segment Jul 27 00:32:55.879252283 (60h57m5.601815323s) [upload=58445]: Waiting for error group managing segments... Jul 27 00:32:56.204275135 (60h57m5.926838175s) [noise=true , address=79.127.205.245:4224 , piece_id=6HOU6RPHEUROKJQOUY6ZQJHBSVHGISDQAULMQW74HFDXL6IOIN2Q, seg_pos={0 0}, upload=58445]: Done uploading piece. err: <nil> Jul 27 00:32:56.240516425 (60h57m5.963079465s) [noise=true , address=79.127.205.246:4273 , piece_id=JDUVUIVGHDAKPHWG6OT5TTUXACYUOZNAUV7LBJUU6A3DXZ4EPFYQ, seg_pos={0 0}, upload=58445]: Done uploading piece. err: <nil> Jul 27 00:32:56.650733016 (60h57m6.373296056s) [noise=true , address=79.127.201.215:4334 , piece_id=VOO7STLQ2BHHYH6S3VOBOV6PA7SRYK7WD6BM4NULH3WNRZ5YX57Q, seg_pos={0 0}, upload=58445]: Done uploading piece. err: <nil> Jul 27 00:32:56.658808818 (60h57m6.381371858s) [noise=true , address=79.127.219.38:4018 , piece_id=APD4R45RKSHB6KJZLS5FI7Z3RBR7W2HAY3PVA6QUZQSOMCVYH44Q, seg_pos={0 0}, upload=58445]: Done uploading piece. err: <nil> Jul 27 00:32:59.443571353 (60h57m9.166134393s) [noise=true , address=79.127.220.101:4041 , piece_id=3JOMURZ4NGLVADZNPRJG6TNUYQN2RYOU55FVMKTXYNKKYUWTBS3A, seg_pos={0 0}, upload=58445]: Done uploading piece. err: <nil> Jul 27 00:33:00.693218114 (60h57m10.415781154s) [noise=true , address=64.49.13.138:4482 , piece_id=2C2ZUJY4IQCVLQZJHAVKSTYHA4Q57SJDIZ4E4XDNAZDIME5Q2DDA, seg_pos={0 0}, upload=58445]: Done uploading piece. err: <nil> Jul 27 00:33:03.497946752 (60h57m13.220509793s) [noise=true , address=45.91.116.14:4194 , piece_id=HWCQ45ZFEEKPTBNTETWJLVZPRTBJQ3VPBH4N66ZN3J5IDIYTUH7Q, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 27 00:33:53.656545686 (60h58m3.379108726s) [noise=true , address=104.36.82.13:4200 , piece_id=E7EWGPINV7NQKERTFIYWGESVCN5AMPKSJ4YHS2YDVCPEGPIUPACQ, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:12dfLumooVsDsWXqi5FQMDutkcexM9A5Kmu4Vo9s1TqfWxv5ozH, address:104.36.82.13:4200): protocol: write tcp 188.166.225.170:39934->104.36.82.13:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.656594209 (60h58m3.379157249s) [noise=true , address=104.36.82.54:4184 , piece_id=WS3QVDC5JQ3HKPJZX6YKR3VRLECQEKF7I5WKZ47ALJE72DEQLKVQ, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG, address:104.36.82.54:4184): protocol: write tcp 188.166.225.170:33688->104.36.82.54:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.656659915 (60h58m3.379222955s) [noise=true , address=77.90.31.13:4653 , piece_id=5BB4YP75W5JS35CVXTFNGJI4KCVQTNLK3X754WTQYDM6UL3TUCJA, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:33:53.656701263 (60h58m3.379264293s) [noise=true , address=104.36.82.52:4118 , piece_id=OMDRZGC7WKORZYQHCHGIMKCSNVGVQTRXRXQW4CMJDNXZLFBTDNYQ, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: rpc: dial tcp 104.36.82.52:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.656743925 (60h58m3.379306965s) [noise=true , address=104.36.82.12:4181 , piece_id=6AAWFKHIIIZ5IYJNEY6B4T3POYNGKZKO5NBPA3ABGXPK7SAVJV2Q, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: rpc: dial tcp 104.36.82.12:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.656887870 (60h58m3.37945093s) [noise=true , address=104.36.82.56:4244 , piece_id=H7IS6EN5K52CPUUV6HMB5IF2O42NUAGGYGBFI25O2SW7ZWOHGNUA, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: rpc: dial tcp 104.36.82.56:4244: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.656919631 (60h58m3.379482671s) [noise=true , address=104.36.82.47:4541 , piece_id=ASRUJWGUUJMPGCT3O34C6PXB2N67WVTBQX4RGJ3WDM5KFGOUQWOA, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.656991669 (60h58m3.379554709s) [noise=true , address=104.36.82.50:4031 , piece_id=67Y72CYO74MH6OR5SN4NYAGDFWWI2VD6LWO4RESH34IX7BVPN2WQ, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: rpc: dial tcp 104.36.82.50:4031: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.657010776 (60h58m3.379573916s) [noise=true , address=77.90.31.21:4908 , piece_id=XXHF5R3QG63IMUYGSKR6LGQ75L5MBIA6P2M4CBR5HSHM52VAT75Q, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: rpc: dial tcp 77.90.31.21:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.657074407 (60h58m3.379637548s) [noise=true , address=104.36.82.46:4499 , piece_id=UQNKWA6VJBN3HGENSJ5MQA3327YUEGJVNS2BQBZCTXZZLGACZ2RA, seg_pos={0 0}, upload=58445]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 188.166.225.170:48758->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:53.657122109 (60h58m3.379685239s) [seg_pos={0 0}, upload=58445]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 27 00:33:53.886272949 (60h58m3.608835989s) [upload=58445]: Done uploading segments... Jul 27 00:33:53.886299741 (60h58m3.608862781s) [upload=58445]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.232447905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXX] duration: 30.254751367s FAIL (2025-07-27 00:33:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.254773579s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [..X...XX..XX.X.XX...X...............XX......X...XXX..XXX..X.] duration: 32.062372639s FAIL (2025-07-27 00:33:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.06238374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.] duration: 7.968439764s FAIL (2025-07-27 00:33:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Jul 27 00:32:43.408103008 (60h56m53.130666048s) [upload=58442]: Starting upload Jul 27 00:32:43.408157693 (60h56m53.130720733s) [upload=58442]: Done starting upload Jul 27 00:32:43.408255180 (60h56m53.13081822s) [upload=58442]: Uploading segments... Jul 27 00:32:43.408328560 (60h56m53.13089159s) [upload=58442]: Got next segment. Inline: false Jul 27 00:32:43.408350582 (60h56m53.130913622s) [upload=58442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:32:43.408354740 (60h56m53.130917781s) [upload=58442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:32:43.881321651 (60h56m53.603884691s) [seg_pos={0 0}, upload=58442]: Begin upload segment... Jul 27 00:32:43.881360195 (60h56m53.603923235s) [seg_pos={0 0}, upload=58442]: Done begin upload segment. Jul 27 00:32:43.881380484 (60h56m53.603943524s) [noise=true , address=77.90.31.19:4842 , piece_id=SSB7BTZ4YP6MIAOLTUF4UOU2YGEGCDZZ63FLFJQ3BXULHNLJOZZA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.881413678 (60h56m53.603976718s) [noise=true , address=45.91.116.14:4191 , piece_id=66SPVSUESWR6QAZXZSHWA2OTMV6K2TRKPPEAVRM72ARR5QKUQZJQ, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.881709604 (60h56m53.604272634s) [noise=true , address=77.90.31.17:4771 , piece_id=VAOGD7J7FIWNR6555IHQDTW7PKOD2VLGQD7FDVWSY2V6OSHAPSLA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.881819825 (60h56m53.604382865s) [noise=true , address=45.91.116.9:4263 , piece_id=K3O4QILW55K2Z7LLINRUF3RRQA7DHS5BC43OBJKTHLPDHUXFXB5A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882068100 (60h56m53.60463114s) [noise=true , address=104.36.82.2:4729 , piece_id=ADOVOXTD3TPSKEHLBHHCWZBEP53CDIIVP6KTEUXHOLQMIO4K3UJA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882093958 (60h56m53.604657009s) [noise=true , address=104.36.82.4:4802 , piece_id=VAPIOATFZAQ5L5VDH2OOYLBJK7GHZ56CXRZR4J7SEXOTWNVIPHTA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882320222 (60h56m53.604883262s) [noise=true , address=45.91.116.11:4118 , piece_id=5RSSPMV4ALEVUHBTBVFO232K35A5XVWWRA4SYUIQ37KON7SQA3JA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882353816 (60h56m53.604916856s) [noise=true , address=77.90.31.16:4733 , piece_id=EED3AZLBDJ6QQQLZ45SQPSXTWIJEY2ZHVQEL7Y75DZIX7ILBVJ4A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882574238 (60h56m53.605137268s) [noise=true , address=104.36.82.8:4969 , piece_id=HEEBQMWV3FD2EFQ6EOFBCBZUGTXRL4Q5XMEAKBCJOLDVT3T2C6XA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882607652 (60h56m53.605170692s) [noise=true , address=45.91.116.5:4298 , piece_id=FCKF25ZLLIXXDQTQGLQI7R4HQXJPZZZGAZKFSCEF7RUDYF3OPYDA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882830197 (60h56m53.605393227s) [noise=true , address=77.90.31.10:4517 , piece_id=MD2S4KRDEQ7LXAZAX3K5GOBTCXBIR3L5Q7SRWAR7CZVCLIINK7EA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882861627 (60h56m53.605424667s) [noise=true , address=104.36.82.4:4803 , piece_id=H6TNKABC23UK2TOTECA4TK2PC3CTPB3X7OUEZ4FCLQLGVUPODB2A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883094243 (60h56m53.605657283s) [noise=true , address=104.36.82.14:4222 , piece_id=MC7SDPFA2VJQZJFVUKJGB7IGV2QCOZQSCEPRGMBFP6ZRJHIMVJ6A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883103470 (60h56m53.60566651s) [noise=true , address=45.91.116.4:4939 , piece_id=JFENBSME4QK7T2AL7DTQWCHK3EDFUVRQ43QKNBM6ZZF4GGGUOOYA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883302541 (60h56m53.605865581s) [noise=true , address=104.36.82.6:4874 , piece_id=5RIK4FFZ7WGJV6DHZ2U2GFDM6WSLQ3UEEIZH4QAFUGCUMK6EKHUQ, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883496001 (60h56m53.606059041s) [noise=true , address=77.90.31.21:4878 , piece_id=DNUT3ZDJ55WTXKETAXOUF2MYEDFNPOO2NNKGGUIWIGMB22TUEJMA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:45.608571945 (60h56m55.331134985s) [upload=58442]: Next returned nil segment Jul 27 00:32:45.608594448 (60h56m55.331157488s) [upload=58442]: Waiting for error group managing segments... Jul 27 00:32:45.900644283 (60h56m55.623207323s) [noise=true , address=77.90.31.10:4517 , piece_id=MD2S4KRDEQ7LXAZAX3K5GOBTCXBIR3L5Q7SRWAR7CZVCLIINK7EA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.903605349 (60h56m55.626168389s) [noise=true , address=45.91.116.5:4298 , piece_id=FCKF25ZLLIXXDQTQGLQI7R4HQXJPZZZGAZKFSCEF7RUDYF3OPYDA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.968895387 (60h56m55.691458428s) [noise=true , address=45.91.116.11:4118 , piece_id=5RSSPMV4ALEVUHBTBVFO232K35A5XVWWRA4SYUIQ37KON7SQA3JA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.984544258 (60h56m55.707107298s) [noise=true , address=77.90.31.16:4733 , piece_id=EED3AZLBDJ6QQQLZ45SQPSXTWIJEY2ZHVQEL7Y75DZIX7ILBVJ4A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.996016899 (60h56m55.718579939s) [noise=true , address=45.91.116.9:4263 , piece_id=K3O4QILW55K2Z7LLINRUF3RRQA7DHS5BC43OBJKTHLPDHUXFXB5A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:46.001325966 (60h56m55.723889057s) [noise=true , address=45.91.116.4:4939 , piece_id=JFENBSME4QK7T2AL7DTQWCHK3EDFUVRQ43QKNBM6ZZF4GGGUOOYA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:46.087716486 (60h56m55.810279526s) [noise=true , address=45.91.116.14:4191 , piece_id=66SPVSUESWR6QAZXZSHWA2OTMV6K2TRKPPEAVRM72ARR5QKUQZJQ, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:47.213006867 (60h56m56.935569907s) [noise=true , address=77.90.31.19:4842 , piece_id=SSB7BTZ4YP6MIAOLTUF4UOU2YGEGCDZZ63FLFJQ3BXULHNLJOZZA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:33:12.763767676 (60h57m22.486330716s) [noise=true , address=77.90.31.17:4771 , piece_id=VAOGD7J7FIWNR6555IHQDTW7PKOD2VLGQD7FDVWSY2V6OSHAPSLA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763815879 (60h57m22.486378919s) [noise=true , address=104.36.82.6:4874 , piece_id=5RIK4FFZ7WGJV6DHZ2U2GFDM6WSLQ3UEEIZH4QAFUGCUMK6EKHUQ, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763845986 (60h57m22.486409036s) [noise=true , address=104.36.82.4:4802 , piece_id=VAPIOATFZAQ5L5VDH2OOYLBJK7GHZ56CXRZR4J7SEXOTWNVIPHTA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763920750 (60h57m22.48648379s) [noise=true , address=104.36.82.14:4222 , piece_id=MC7SDPFA2VJQZJFVUKJGB7IGV2QCOZQSCEPRGMBFP6ZRJHIMVJ6A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763959964 (60h57m22.486523004s) [noise=true , address=104.36.82.4:4803 , piece_id=H6TNKABC23UK2TOTECA4TK2PC3CTPB3X7OUEZ4FCLQLGVUPODB2A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763971887 (60h57m22.486534928s) [noise=true , address=77.90.31.21:4878 , piece_id=DNUT3ZDJ55WTXKETAXOUF2MYEDFNPOO2NNKGGUIWIGMB22TUEJMA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763991484 (60h57m22.486554514s) [noise=true , address=104.36.82.2:4729 , piece_id=ADOVOXTD3TPSKEHLBHHCWZBEP53CDIIVP6KTEUXHOLQMIO4K3UJA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:59402->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.764000212 (60h57m22.486563252s) [noise=true , address=104.36.82.8:4969 , piece_id=HEEBQMWV3FD2EFQ6EOFBCBZUGTXRL4Q5XMEAKBCJOLDVT3T2C6XA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:46146->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.764016813 (60h57m22.486579853s) [seg_pos={0 0}, upload=58442]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:33:12.996500200 (60h57m22.71906324s) [upload=58442]: Done uploading segments... Jul 27 00:33:12.996530999 (60h57m22.719094089s) [upload=58442]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.968481183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX] duration: 30.235010127s FAIL (2025-07-27 00:33:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Jul 27 00:32:43.408103008 (60h56m53.130666048s) [upload=58442]: Starting upload Jul 27 00:32:43.408157693 (60h56m53.130720733s) [upload=58442]: Done starting upload Jul 27 00:32:43.408255180 (60h56m53.13081822s) [upload=58442]: Uploading segments... Jul 27 00:32:43.408328560 (60h56m53.13089159s) [upload=58442]: Got next segment. Inline: false Jul 27 00:32:43.408350582 (60h56m53.130913622s) [upload=58442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:32:43.408354740 (60h56m53.130917781s) [upload=58442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:32:43.881321651 (60h56m53.603884691s) [seg_pos={0 0}, upload=58442]: Begin upload segment... Jul 27 00:32:43.881360195 (60h56m53.603923235s) [seg_pos={0 0}, upload=58442]: Done begin upload segment. Jul 27 00:32:43.881380484 (60h56m53.603943524s) [noise=true , address=77.90.31.19:4842 , piece_id=SSB7BTZ4YP6MIAOLTUF4UOU2YGEGCDZZ63FLFJQ3BXULHNLJOZZA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.881413678 (60h56m53.603976718s) [noise=true , address=45.91.116.14:4191 , piece_id=66SPVSUESWR6QAZXZSHWA2OTMV6K2TRKPPEAVRM72ARR5QKUQZJQ, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.881709604 (60h56m53.604272634s) [noise=true , address=77.90.31.17:4771 , piece_id=VAOGD7J7FIWNR6555IHQDTW7PKOD2VLGQD7FDVWSY2V6OSHAPSLA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.881819825 (60h56m53.604382865s) [noise=true , address=45.91.116.9:4263 , piece_id=K3O4QILW55K2Z7LLINRUF3RRQA7DHS5BC43OBJKTHLPDHUXFXB5A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882068100 (60h56m53.60463114s) [noise=true , address=104.36.82.2:4729 , piece_id=ADOVOXTD3TPSKEHLBHHCWZBEP53CDIIVP6KTEUXHOLQMIO4K3UJA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882093958 (60h56m53.604657009s) [noise=true , address=104.36.82.4:4802 , piece_id=VAPIOATFZAQ5L5VDH2OOYLBJK7GHZ56CXRZR4J7SEXOTWNVIPHTA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882320222 (60h56m53.604883262s) [noise=true , address=45.91.116.11:4118 , piece_id=5RSSPMV4ALEVUHBTBVFO232K35A5XVWWRA4SYUIQ37KON7SQA3JA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882353816 (60h56m53.604916856s) [noise=true , address=77.90.31.16:4733 , piece_id=EED3AZLBDJ6QQQLZ45SQPSXTWIJEY2ZHVQEL7Y75DZIX7ILBVJ4A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882574238 (60h56m53.605137268s) [noise=true , address=104.36.82.8:4969 , piece_id=HEEBQMWV3FD2EFQ6EOFBCBZUGTXRL4Q5XMEAKBCJOLDVT3T2C6XA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882607652 (60h56m53.605170692s) [noise=true , address=45.91.116.5:4298 , piece_id=FCKF25ZLLIXXDQTQGLQI7R4HQXJPZZZGAZKFSCEF7RUDYF3OPYDA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882830197 (60h56m53.605393227s) [noise=true , address=77.90.31.10:4517 , piece_id=MD2S4KRDEQ7LXAZAX3K5GOBTCXBIR3L5Q7SRWAR7CZVCLIINK7EA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.882861627 (60h56m53.605424667s) [noise=true , address=104.36.82.4:4803 , piece_id=H6TNKABC23UK2TOTECA4TK2PC3CTPB3X7OUEZ4FCLQLGVUPODB2A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883094243 (60h56m53.605657283s) [noise=true , address=104.36.82.14:4222 , piece_id=MC7SDPFA2VJQZJFVUKJGB7IGV2QCOZQSCEPRGMBFP6ZRJHIMVJ6A, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883103470 (60h56m53.60566651s) [noise=true , address=45.91.116.4:4939 , piece_id=JFENBSME4QK7T2AL7DTQWCHK3EDFUVRQ43QKNBM6ZZF4GGGUOOYA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883302541 (60h56m53.605865581s) [noise=true , address=104.36.82.6:4874 , piece_id=5RIK4FFZ7WGJV6DHZ2U2GFDM6WSLQ3UEEIZH4QAFUGCUMK6EKHUQ, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:43.883496001 (60h56m53.606059041s) [noise=true , address=77.90.31.21:4878 , piece_id=DNUT3ZDJ55WTXKETAXOUF2MYEDFNPOO2NNKGGUIWIGMB22TUEJMA, seg_pos={0 0}, upload=58442]: Uploading piece... Jul 27 00:32:45.608571945 (60h56m55.331134985s) [upload=58442]: Next returned nil segment Jul 27 00:32:45.608594448 (60h56m55.331157488s) [upload=58442]: Waiting for error group managing segments... Jul 27 00:32:45.900644283 (60h56m55.623207323s) [noise=true , address=77.90.31.10:4517 , piece_id=MD2S4KRDEQ7LXAZAX3K5GOBTCXBIR3L5Q7SRWAR7CZVCLIINK7EA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.903605349 (60h56m55.626168389s) [noise=true , address=45.91.116.5:4298 , piece_id=FCKF25ZLLIXXDQTQGLQI7R4HQXJPZZZGAZKFSCEF7RUDYF3OPYDA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.968895387 (60h56m55.691458428s) [noise=true , address=45.91.116.11:4118 , piece_id=5RSSPMV4ALEVUHBTBVFO232K35A5XVWWRA4SYUIQ37KON7SQA3JA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.984544258 (60h56m55.707107298s) [noise=true , address=77.90.31.16:4733 , piece_id=EED3AZLBDJ6QQQLZ45SQPSXTWIJEY2ZHVQEL7Y75DZIX7ILBVJ4A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:45.996016899 (60h56m55.718579939s) [noise=true , address=45.91.116.9:4263 , piece_id=K3O4QILW55K2Z7LLINRUF3RRQA7DHS5BC43OBJKTHLPDHUXFXB5A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:46.001325966 (60h56m55.723889057s) [noise=true , address=45.91.116.4:4939 , piece_id=JFENBSME4QK7T2AL7DTQWCHK3EDFUVRQ43QKNBM6ZZF4GGGUOOYA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:46.087716486 (60h56m55.810279526s) [noise=true , address=45.91.116.14:4191 , piece_id=66SPVSUESWR6QAZXZSHWA2OTMV6K2TRKPPEAVRM72ARR5QKUQZJQ, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:32:47.213006867 (60h56m56.935569907s) [noise=true , address=77.90.31.19:4842 , piece_id=SSB7BTZ4YP6MIAOLTUF4UOU2YGEGCDZZ63FLFJQ3BXULHNLJOZZA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: <nil> Jul 27 00:33:12.763767676 (60h57m22.486330716s) [noise=true , address=77.90.31.17:4771 , piece_id=VAOGD7J7FIWNR6555IHQDTW7PKOD2VLGQD7FDVWSY2V6OSHAPSLA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763815879 (60h57m22.486378919s) [noise=true , address=104.36.82.6:4874 , piece_id=5RIK4FFZ7WGJV6DHZ2U2GFDM6WSLQ3UEEIZH4QAFUGCUMK6EKHUQ, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763845986 (60h57m22.486409036s) [noise=true , address=104.36.82.4:4802 , piece_id=VAPIOATFZAQ5L5VDH2OOYLBJK7GHZ56CXRZR4J7SEXOTWNVIPHTA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763920750 (60h57m22.48648379s) [noise=true , address=104.36.82.14:4222 , piece_id=MC7SDPFA2VJQZJFVUKJGB7IGV2QCOZQSCEPRGMBFP6ZRJHIMVJ6A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763959964 (60h57m22.486523004s) [noise=true , address=104.36.82.4:4803 , piece_id=H6TNKABC23UK2TOTECA4TK2PC3CTPB3X7OUEZ4FCLQLGVUPODB2A, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763971887 (60h57m22.486534928s) [noise=true , address=77.90.31.21:4878 , piece_id=DNUT3ZDJ55WTXKETAXOUF2MYEDFNPOO2NNKGGUIWIGMB22TUEJMA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.763991484 (60h57m22.486554514s) [noise=true , address=104.36.82.2:4729 , piece_id=ADOVOXTD3TPSKEHLBHHCWZBEP53CDIIVP6KTEUXHOLQMIO4K3UJA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:59402->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.764000212 (60h57m22.486563252s) [noise=true , address=104.36.82.8:4969 , piece_id=HEEBQMWV3FD2EFQ6EOFBCBZUGTXRL4Q5XMEAKBCJOLDVT3T2C6XA, seg_pos={0 0}, upload=58442]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:46146->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:12.764016813 (60h57m22.486579853s) [seg_pos={0 0}, upload=58442]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:33:12.996500200 (60h57m22.71906324s) [upload=58442]: Done uploading segments... Jul 27 00:33:12.996530999 (60h57m22.719094089s) [upload=58442]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235030093s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [...X...XX..XX.X.XX...X...............XX......X...XXX..XXX..X] duration: 1m0.001067498s FAIL (2025-07-27 00:32:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001078169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXX] duration: 30.239189543s FAIL (2025-07-27 00:32:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.239204764s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................................X..........] duration: 1.855873416s FAIL (2025-07-27 00:32:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.855901791s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231048203s FAIL (2025-07-27 00:32:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 59758 (60h55m57.785822798s) [seg_pos={0 0}, upload=58425]: Begin upload segment... Jul 27 00:31:48.063313300 (60h55m57.78587636s) [seg_pos={0 0}, upload=58425]: Done begin upload segment. Jul 27 00:31:48.063371291 (60h55m57.785934362s) [noise=true , address=79.127.205.235:4668 , piece_id=ZAYWYU45CGZQYBTXCG4ISUJOFY6EU74RLAXEVF4CTI6RDDSCKAEA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.063978744 (60h55m57.786541814s) [noise=true , address=77.90.31.10:4525 , piece_id=AI7FNR5HE3I2HPYV4CMX5HPJZMWYY7YWTSKRK256IXZEX2CQ3WWQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.064147566 (60h55m57.786710647s) [noise=true , address=79.127.205.234:4619 , piece_id=ZYGZJGCMUJUFWGQIHEPKPLJKLWCPYAVEZPPO4DEZY4YZU3F5QGUA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.064746332 (60h55m57.787309382s) [noise=true , address=104.36.82.45:4905 , piece_id=3VNFKUGFDAXJP5FKTW4PKS332HGZ2CVYZNK6TIMNO3RB7HPW7LNQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.064960471 (60h55m57.787523521s) [noise=true , address=64.49.13.138:4482 , piece_id=VPFDUZAJ4JU7MWN4RSA7G7RFBODXIZELEHGJ7D3PYUPEXHDPZZUQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.065232311 (60h55m57.787795362s) [noise=true , address=77.90.31.18:4817 , piece_id=HXZ34R6V5ORDXQIURA52QGDNEFBWZ2G6IB5KOZN2TKOVBYX6E2YQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.065393390 (60h55m57.78795644s) [noise=true , address=104.36.82.54:4190 , piece_id=LJCZRAZNSUPWSDB7JBGOX5TQREOQNX6RGJDVFLYLB45CDCRVJ5TQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.065718452 (60h55m57.788281492s) [noise=true , address=104.36.82.47:4524 , piece_id=IU52HBGQYJSUYFWT5W5STOH6JTBHR6YHNUHVV4H5HTVEXAGB2CDA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.065862387 (60h55m57.788425437s) [noise=true , address=64.49.13.138:4483 , piece_id=WJOKKGLBDVNNBADVMRVPFMA4P4MMMYQZT63IDMCUDOW3AHISX2QQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.066224079 (60h55m57.788787119s) [noise=true , address=79.127.205.233:4656 , piece_id=HNRQYAP4BOM4P4PJSS27YBSJBI5VA2EY26HKXJFIHY6BEVRCMP3Q, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.066343197 (60h55m57.788906238s) [noise=true , address=104.36.82.46:4483 , piece_id=C3QEE2AC2DQMWLNRIGWUFUFZG65EOB7BSYVBM4IEIG3YY3YXNPDA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.066701934 (60h55m57.789264974s) [noise=true , address=79.127.201.215:4347 , piece_id=VUIMDKITZSREJZUAYXJYTKPZAE632SJGFAK5EW4OAG3FQ2NYZZIQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.066711632 (60h55m57.789274672s) [noise=true , address=104.36.82.52:4098 , piece_id=4NHDQX3KJOWF35YODWW6WJLC2IS27Q3KBL6KSBTX3KAYLLOGXN5A, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.067010053 (60h55m57.789573093s) [noise=true , address=77.90.31.16:4754 , piece_id=GPPTQZUGGMQUV42GXNSGUH22W5YSUE6ULMJH4SNPT4QUBK6I6ZIA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.067055841 (60h55m57.789618881s) [noise=true , address=104.36.82.55:4205 , piece_id=KXLEQ2ACYGAI2QQJHHPX2LZ2JLF6GATXAGJ2C57IPQZP2AUOAU7Q, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.067387906 (60h55m57.789950946s) [noise=true , address=77.90.31.13:4644 , piece_id=4AAYEXK3XYONMJCHII2TCRQEYCMZJ6Y7W34BDRTGZLAV7F45MCYA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.067433503 (60h55m57.789996543s) [noise=true , address=104.36.82.53:4130 , piece_id=V5K4HMXLCZVGYPBNM4GZ6LEMVSJ3GQFLFNOQTPMJXPRVWC45EU2A, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:48.067737566 (60h55m57.790300596s) [noise=true , address=45.91.116.8:4062 , piece_id=HILHUQXNPXTZZT5DU5H7H7FTWKRCQRYV2SK6KYCQVUNFFLSWXRPA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:31:49.177988459 (60h55m58.900551499s) [upload=58425]: Next returned nil segment Jul 27 00:31:49.178005121 (60h55m58.900568161s) [upload=58425]: Waiting for error group managing segments... Jul 27 00:31:49.465296836 (60h55m59.187859876s) [noise=true , address=79.127.205.233:4656 , piece_id=HNRQYAP4BOM4P4PJSS27YBSJBI5VA2EY26HKXJFIHY6BEVRCMP3Q, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:49.469243267 (60h55m59.191806307s) [noise=true , address=79.127.205.234:4619 , piece_id=ZYGZJGCMUJUFWGQIHEPKPLJKLWCPYAVEZPPO4DEZY4YZU3F5QGUA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:49.476259781 (60h55m59.198822821s) [noise=true , address=79.127.205.235:4668 , piece_id=ZAYWYU45CGZQYBTXCG4ISUJOFY6EU74RLAXEVF4CTI6RDDSCKAEA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:50.014059675 (60h55m59.736622715s) [noise=true , address=45.91.116.8:4062 , piece_id=HILHUQXNPXTZZT5DU5H7H7FTWKRCQRYV2SK6KYCQVUNFFLSWXRPA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:50.079097701 (60h55m59.801660742s) [noise=true , address=77.90.31.16:4754 , piece_id=GPPTQZUGGMQUV42GXNSGUH22W5YSUE6ULMJH4SNPT4QUBK6I6ZIA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:51.154070387 (60h56m0.876633427s) [noise=true , address=64.49.13.138:4483 , piece_id=WJOKKGLBDVNNBADVMRVPFMA4P4MMMYQZT63IDMCUDOW3AHISX2QQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:54.427278369 (60h56m4.149841409s) [noise=true , address=64.49.13.138:4482 , piece_id=VPFDUZAJ4JU7MWN4RSA7G7RFBODXIZELEHGJ7D3PYUPEXHDPZZUQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:54.471312310 (60h56m4.19387535s) [noise=true , address=77.90.31.13:4644 , piece_id=4AAYEXK3XYONMJCHII2TCRQEYCMZJ6Y7W34BDRTGZLAV7F45MCYA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:31:57.703708817 (60h56m7.426271857s) [noise=true , address=79.127.201.215:4347 , piece_id=VUIMDKITZSREJZUAYXJYTKPZAE632SJGFAK5EW4OAG3FQ2NYZZIQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:05.453068331 (60h56m15.175631371s) [noise=true , address=77.90.31.10:4525 , piece_id=AI7FNR5HE3I2HPYV4CMX5HPJZMWYY7YWTSKRK256IXZEX2CQ3WWQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:09.365496342 (60h56m19.088059382s) [noise=true , address=77.90.31.18:4817 , piece_id=HXZ34R6V5ORDXQIURA52QGDNEFBWZ2G6IB5KOZN2TKOVBYX6E2YQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:46.984929401 (60h56m56.707492441s) [noise=true , address=104.36.82.55:4205 , piece_id=KXLEQ2ACYGAI2QQJHHPX2LZ2JLF6GATXAGJ2C57IPQZP2AUOAU7Q, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:104.36.82.55:4205): protocol: rpc: dial tcp 104.36.82.55:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.984999674 (60h56m56.707562715s) [noise=true , address=104.36.82.53:4130 , piece_id=V5K4HMXLCZVGYPBNM4GZ6LEMVSJ3GQFLFNOQTPMJXPRVWC45EU2A, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: rpc: dial tcp 104.36.82.53:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.985035223 (60h56m56.707598263s) [noise=true , address=104.36.82.46:4483 , piece_id=C3QEE2AC2DQMWLNRIGWUFUFZG65EOB7BSYVBM4IEIG3YY3YXNPDA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: rpc: dial tcp 104.36.82.46:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.985095639 (60h56m56.707658689s) [noise=true , address=104.36.82.45:4905 , piece_id=3VNFKUGFDAXJP5FKTW4PKS332HGZ2CVYZNK6TIMNO3RB7HPW7LNQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: rpc: dial tcp 104.36.82.45:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.985133090 (60h56m56.70769613s) [noise=true , address=104.36.82.54:4190 , piece_id=LJCZRAZNSUPWSDB7JBGOX5TQREOQNX6RGJDVFLYLB45CDCRVJ5TQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.985187314 (60h56m56.707750354s) [noise=true , address=104.36.82.47:4524 , piece_id=IU52HBGQYJSUYFWT5W5STOH6JTBHR6YHNUHVV4H5HTVEXAGB2CDA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.985247899 (60h56m56.707810939s) [noise=true , address=104.36.82.52:4098 , piece_id=4NHDQX3KJOWF35YODWW6WJLC2IS27Q3KBL6KSBTX3KAYLLOGXN5A, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1SoFNdwfh4p5tXUkpfjQaev6tdA2ZdhsXUFvZNBsy5QrJ9zdqb, address:104.36.82.52:4098): protocol: write tcp 188.166.225.170:40798->104.36.82.52:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:46.985264241 (60h56m56.707827281s) [seg_pos={0 0}, upload=58425]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:32:47.213228772 (60h56m56.935791812s) [upload=58425]: Done uploading segments... Jul 27 00:32:47.213265141 (60h56m56.935828181s) [upload=58425]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.231058102s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..X..] duration: 5.930809052s FAIL (2025-07-27 00:32:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.930829159s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXX] duration: 30.229316405s FAIL (2025-07-27 00:32:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== n closed: op Jul 27 00:31:37.372189485 (60h55m47.094752536s) [noise=true , address=104.36.82.6:4875 , piece_id=UNO44YFT6JQDXB2W5JULL56SS35KHD7JPMAHGZVE6W53FZAOODPQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:36286->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:37.372203292 (60h55m47.094766332s) [seg_pos={0 0}, upload=58414]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:31:37.606197684 (60h55m47.328760724s) [upload=58414]: Done uploading segments... Jul 27 00:31:37.606223393 (60h55m47.328786433s) [upload=58414]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:32:03.581458881 (60h56m13.304021921s) [upload=58432]: Starting upload Jul 27 00:32:03.581523194 (60h56m13.304086234s) [upload=58432]: Done starting upload Jul 27 00:32:03.581593248 (60h56m13.304156288s) [upload=58432]: Uploading segments... Jul 27 00:32:03.581716373 (60h56m13.304279413s) [upload=58432]: Got next segment. Inline: false Jul 27 00:32:03.581746471 (60h56m13.304309511s) [upload=58432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:32:03.581751009 (60h56m13.304314049s) [upload=58432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:32:03.869825693 (60h56m13.592388733s) [seg_pos={0 0}, upload=58432]: Begin upload segment... Jul 27 00:32:03.869862935 (60h56m13.592425975s) [seg_pos={0 0}, upload=58432]: Done begin upload segment. Jul 27 00:32:03.869884154 (60h56m13.592447195s) [noise=true , address=77.90.31.15:4698 , piece_id=I3KW2X3YGBDQ24RM7YFLR5UZVL3XVKWZK6M5PCREEBGFRIFGESDQ, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.869933239 (60h56m13.592496279s) [noise=true , address=45.91.116.12:4227 , piece_id=ICPCVZPZ6N2KSCY6XDCK24T23MG4S4QSPT7LH2TQX44ZE2TGQE7A, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.870236599 (60h56m13.592799629s) [noise=true , address=45.91.116.14:4191 , piece_id=KPRAS7AU4RDJNKR5LPHX3IN6OJKSPJUXTKTZM3BY4A7GIF2XEGBA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.870336280 (60h56m13.59289931s) [noise=true , address=45.91.116.5:4299 , piece_id=ZCXYPLOTX6CBMSXYZ5M3OUWH3PHHW5PBIJLBRZMOCNDCBFEZJQZA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.870490935 (60h56m13.593053965s) [noise=true , address=104.36.82.16:4295 , piece_id=55WJLKXE5NBO22CMSCA2FE5D3HPXQ73V3J6WLOB4L27V72A5B54A, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.870642305 (60h56m13.593205346s) [noise=true , address=45.91.116.10:4081 , piece_id=FMEUKG6CNZWQBM32MWHD2QYPXXBJCS76YMJ7TTABUWX3R55KF3ZA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.870769169 (60h56m13.593332239s) [noise=true , address=45.91.116.4:4937 , piece_id=G4ZPGOBA2YTHL45V3KFYNJQQZMAIB6EC5JOADE4PZO36JSK4SX4A, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.870902353 (60h56m13.593465393s) [noise=true , address=77.90.31.14:4663 , piece_id=OBMIXH25PSFLJ4EG3MQIO2FWMYJGO7YUWBNYW2BPQQTCYCGPDA7A, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871038934 (60h56m13.593601974s) [noise=true , address=77.90.31.19:4842 , piece_id=E6UIYHCAAISCVUWFVIOAZ4LNYL2V42QVKHECM7USMYEDYV3JSHZA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871170175 (60h56m13.593733205s) [noise=true , address=104.36.82.10:4077 , piece_id=IGKVPJ5D3K5CTKSBMROHMPHM75ISV3DFWESR3MUAWPCVVNWNTLLA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871296317 (60h56m13.593859347s) [noise=true , address=77.90.31.13:4627 , piece_id=KMUY7F6VQRWZHFKP7L7CG4NCNHUVSQE22WQSS63CXLYKRI4W6OGA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871386820 (60h56m13.59394986s) [noise=true , address=45.91.116.6:4973 , piece_id=5CFTWWQ2YHT5AFNFO5BB4FZC4UGL4SHSRJTGDMPAH3GXXCTWMQTA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871505937 (60h56m13.594068977s) [noise=true , address=77.90.31.13:4628 , piece_id=5G6AZFFKWWIW7FTAO2JYST4NQXEDV2KQFKOQMJSBFXPLYI7UYDAQ, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871602262 (60h56m13.594165302s) [noise=true , address=45.91.116.8:4045 , piece_id=FP2PDOBY3FY6HGDS3GJ56R7JWAOCJO7BFVIWWQQRLKUIPKNOMUZQ, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871705119 (60h56m13.594268159s) [noise=true , address=77.90.31.12:4589 , piece_id=NN7MY6V7CPJ5RKFPLUL75ROYB35MLWBJBGVGIPRQZYRCAY6BLC7A, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:03.871822263 (60h56m13.594385303s) [noise=true , address=104.36.82.15:4259 , piece_id=B4LWL5Y2YIGXB7YCE7TBGRJH5R3EEBNGPLWABNJC4QYDTXOABKXA, seg_pos={0 0}, upload=58432]: Uploading piece... Jul 27 00:32:05.606450689 (60h56m15.329013729s) [upload=58432]: Next returned nil segment Jul 27 00:32:05.606459625 (60h56m15.329022655s) [upload=58432]: Waiting for error group managing segments... Jul 27 00:32:05.886265828 (60h56m15.608828868s) [noise=true , address=45.91.116.12:4227 , piece_id=ICPCVZPZ6N2KSCY6XDCK24T23MG4S4QSPT7LH2TQX44ZE2TGQE7A, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:05.914574802 (60h56m15.637137832s) [noise=true , address=77.90.31.14:4663 , piece_id=OBMIXH25PSFLJ4EG3MQIO2FWMYJGO7YUWBNYW2BPQQTCYCGPDA7A, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:05.963699552 (60h56m15.686262582s) [noise=true , address=45.91.116.8:4045 , piece_id=FP2PDOBY3FY6HGDS3GJ56R7JWAOCJO7BFVIWWQQRLKUIPKNOMUZQ, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:05.982968349 (60h56m15.705531389s) [noise=true , address=45.91.116.6:4973 , piece_id=5CFTWWQ2YHT5AFNFO5BB4FZC4UGL4SHSRJTGDMPAH3GXXCTWMQTA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:06.004120511 (60h56m15.726683551s) [noise=true , address=45.91.116.5:4299 , piece_id=ZCXYPLOTX6CBMSXYZ5M3OUWH3PHHW5PBIJLBRZMOCNDCBFEZJQZA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:06.010461672 (60h56m15.733024712s) [noise=true , address=45.91.116.14:4191 , piece_id=KPRAS7AU4RDJNKR5LPHX3IN6OJKSPJUXTKTZM3BY4A7GIF2XEGBA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:06.045191882 (60h56m15.767754922s) [noise=true , address=45.91.116.10:4081 , piece_id=FMEUKG6CNZWQBM32MWHD2QYPXXBJCS76YMJ7TTABUWX3R55KF3ZA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:06.063568502 (60h56m15.786131542s) [noise=true , address=45.91.116.4:4937 , piece_id=G4ZPGOBA2YTHL45V3KFYNJQQZMAIB6EC5JOADE4PZO36JSK4SX4A, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:07.311097319 (60h56m17.033660349s) [noise=true , address=77.90.31.19:4842 , piece_id=E6UIYHCAAISCVUWFVIOAZ4LNYL2V42QVKHECM7USMYEDYV3JSHZA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:07.313706401 (60h56m17.036269441s) [noise=true , address=77.90.31.13:4628 , piece_id=5G6AZFFKWWIW7FTAO2JYST4NQXEDV2KQFKOQMJSBFXPLYI7UYDAQ, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:07.422655675 (60h56m17.145218705s) [noise=true , address=77.90.31.12:4589 , piece_id=NN7MY6V7CPJ5RKFPLUL75ROYB35MLWBJBGVGIPRQZYRCAY6BLC7A, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:08.067313648 (60h56m17.789876688s) [noise=true , address=77.90.31.13:4627 , piece_id=KMUY7F6VQRWZHFKP7L7CG4NCNHUVSQE22WQSS63CXLYKRI4W6OGA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: <nil> Jul 27 00:32:32.938700584 (60h56m42.661263624s) [noise=true , address=104.36.82.16:4295 , piece_id=55WJLKXE5NBO22CMSCA2FE5D3HPXQ73V3J6WLOB4L27V72A5B54A, seg_pos={0 0}, upload=58432]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.938736312 (60h56m42.661299353s) [noise=true , address=104.36.82.10:4077 , piece_id=IGKVPJ5D3K5CTKSBMROHMPHM75ISV3DFWESR3MUAWPCVVNWNTLLA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:46566->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.938768724 (60h56m42.661331764s) [noise=true , address=104.36.82.15:4259 , piece_id=B4LWL5Y2YIGXB7YCE7TBGRJH5R3EEBNGPLWABNJC4QYDTXOABKXA, seg_pos={0 0}, upload=58432]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:54774->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.938780177 (60h56m42.661343217s) [noise=true , address=77.90.31.15:4698 , piece_id=I3KW2X3YGBDQ24RM7YFLR5UZVL3XVKWZK6M5PCREEBGFRIFGESDQ, seg_pos={0 0}, upload=58432]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.938799623 (60h56m42.661362663s) [seg_pos={0 0}, upload=58432]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:32:33.166123543 (60h56m42.888686583s) [upload=58432]: Done uploading segments... Jul 27 00:32:33.166148110 (60h56m42.88871115s) [upload=58432]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.229339019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XX] duration: 30.232310205s FAIL (2025-07-27 00:32:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.232332417s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..X.] duration: 20.88808884s FAIL (2025-07-27 00:32:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.888097777s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................................X.........] duration: 1.893795131s FAIL (2025-07-27 00:32:18 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.893828566s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....X...XX..XX.X.XX...X...............XX......X...XXX..XXX..] duration: 47.263233121s FAIL (2025-07-27 00:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.263249053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..X] duration: 30.232584396s FAIL (2025-07-27 00:31:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232606569s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228478208s FAIL (2025-07-27 00:31:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 30:47.173850141 (60h54m56.896413171s) [upload=58409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:30:47.614869062 (60h54m57.337432102s) [seg_pos={0 0}, upload=58409]: Begin upload segment... Jul 27 00:30:47.614919138 (60h54m57.337482178s) [seg_pos={0 0}, upload=58409]: Done begin upload segment. Jul 27 00:30:47.614939687 (60h54m57.337502717s) [noise=true , address=77.90.31.12:4599 , piece_id=SOX3KNYYTNRJUVT7WNLRXU3JTDN2GGDJQFA3S2WU55Q6YZ452XZA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.614984393 (60h54m57.337547433s) [noise=true , address=104.36.82.55:4195 , piece_id=VXL3FSQ3B5ILDC24DIXJSQH3GNUHL5WN2KONCPB3WEBNC4GPBJSA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.615283465 (60h54m57.337846495s) [noise=true , address=79.127.205.245:4225 , piece_id=LSR4XEP3XB6743SWQ62GUGTFNXXXNRC7WZXWOTZ26WLT22TVANPA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.615487856 (60h54m57.338050896s) [noise=true , address=79.127.205.234:4612 , piece_id=NV3RODH6PTNKFYSPFBJVC37LQIYVXD23NBBD2Y22FKD4SH27I65Q, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.615554464 (60h54m57.338117504s) [noise=true , address=79.127.219.38:4003 , piece_id=BZNW7VU65R4JSIAUOTVNGARSSEXLWUJAR3KCWYXEFHSHMOMIJI7A, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.615993133 (60h54m57.338556173s) [noise=true , address=104.36.82.53:4135 , piece_id=MBVOXZ2YFD4UFXSQJEDN6L745ET7HLK6ILOBN2I3EGZC6UNUZZYA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.616001409 (60h54m57.338564449s) [noise=true , address=104.36.82.47:4528 , piece_id=5EWLBC4WA5G4JMVFOBUKUSVJAUCOP5TCYQB6WVMARPLCR7IDCPGQ, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.616270915 (60h54m57.338833955s) [noise=true , address=77.90.31.19:4847 , piece_id=7KAEK5CWYZQ5N4TWTYPEDLFQ7BJWJZ4V56FTMWQI2WK4IER3C75A, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.616529098 (60h54m57.339092138s) [noise=true , address=79.127.205.246:4284 , piece_id=75VA6O3DTCWLINBTYNWCURQI5LI27JT4DP6XRUUJMHG35WGGGBYA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.616753317 (60h54m57.339316357s) [noise=true , address=77.90.31.10:4537 , piece_id=KCMNHZI24YLC7ZCGYL3O3KK5DLSCZN3OYZED6C5ZWOO4EHW5RDGQ, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.616953871 (60h54m57.339516912s) [noise=true , address=104.36.82.12:4153 , piece_id=23KWNTGZFLYY7TD35DUKAQ6EB7CZM4Z6PER7BCEEUOD2N5WN7TWA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.617155597 (60h54m57.339718627s) [noise=true , address=77.90.31.21:4905 , piece_id=DU6JFGT5CPG2FZOQ66KAN54FJFJP23I6NMRGYBT2PYQBH2DZ3DPQ, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.617360750 (60h54m57.33992379s) [noise=true , address=104.36.82.46:4487 , piece_id=V4DJD45D6XKXJACFI2BX5BU6PAOZTQGBW36SFT7KK5IT4ZSVUY5A, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.617568908 (60h54m57.340131938s) [noise=true , address=45.91.116.9:4277 , piece_id=MRCVEKCWUY4XAJJK3UQBEYVBDOXBE3VYX75ZPQX5DBTKBLDWIVQQ, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.617752069 (60h54m57.340315109s) [noise=true , address=77.90.31.13:4634 , piece_id=X6DJQMEQJ4PMFRKXZG2QTFDJX7XHWEAGMCR7YARIMNMG4FT5DJ7Q, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.617959545 (60h54m57.340522585s) [noise=true , address=64.49.13.138:4483 , piece_id=6OAHEZGK7CPOQLBXETQYHPO5DW74HLCL535JARLN6KFYYZGA5P6A, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.618143848 (60h54m57.340706878s) [noise=true , address=79.127.220.101:4041 , piece_id=P2MHNERVA2EP6XZXXUUY7DHQQIK5RGKHVIOJFKTGFCMCIBWW5AWA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:47.618382074 (60h54m57.340945114s) [noise=true , address=79.127.205.235:4676 , piece_id=ZHHZ6BZDTTW3T4KRF6U3GTY65ZLLPBZPREO3IT2CZWJUUTHCD6GA, seg_pos={0 0}, upload=58409]: Uploading piece... Jul 27 00:30:48.713309839 (60h54m58.435872879s) [upload=58409]: Next returned nil segment Jul 27 00:30:48.713325349 (60h54m58.435888389s) [upload=58409]: Waiting for error group managing segments... Jul 27 00:30:49.003644517 (60h54m58.726207557s) [noise=true , address=79.127.205.246:4284 , piece_id=75VA6O3DTCWLINBTYNWCURQI5LI27JT4DP6XRUUJMHG35WGGGBYA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:49.003696406 (60h54m58.726259446s) [noise=true , address=79.127.205.245:4225 , piece_id=LSR4XEP3XB6743SWQ62GUGTFNXXXNRC7WZXWOTZ26WLT22TVANPA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:49.041957612 (60h54m58.764520652s) [noise=true , address=79.127.205.234:4612 , piece_id=NV3RODH6PTNKFYSPFBJVC37LQIYVXD23NBBD2Y22FKD4SH27I65Q, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:49.115600823 (60h54m58.838163863s) [noise=true , address=79.127.205.235:4676 , piece_id=ZHHZ6BZDTTW3T4KRF6U3GTY65ZLLPBZPREO3IT2CZWJUUTHCD6GA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:49.207059821 (60h54m58.929622861s) [noise=true , address=45.91.116.9:4277 , piece_id=MRCVEKCWUY4XAJJK3UQBEYVBDOXBE3VYX75ZPQX5DBTKBLDWIVQQ, seg_pos={0 0}, upload=58409]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:30:49.622641654 (60h54m59.345204694s) [noise=true , address=79.127.219.38:4003 , piece_id=BZNW7VU65R4JSIAUOTVNGARSSEXLWUJAR3KCWYXEFHSHMOMIJI7A, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:49.653473184 (60h54m59.376036224s) [noise=true , address=77.90.31.10:4537 , piece_id=KCMNHZI24YLC7ZCGYL3O3KK5DLSCZN3OYZED6C5ZWOO4EHW5RDGQ, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:49.698028993 (60h54m59.420592033s) [noise=true , address=77.90.31.21:4905 , piece_id=DU6JFGT5CPG2FZOQ66KAN54FJFJP23I6NMRGYBT2PYQBH2DZ3DPQ, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:50.222473363 (60h54m59.945036403s) [noise=true , address=79.127.220.101:4041 , piece_id=P2MHNERVA2EP6XZXXUUY7DHQQIK5RGKHVIOJFKTGFCMCIBWW5AWA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:55.933959876 (60h55m5.656522916s) [noise=true , address=77.90.31.19:4847 , piece_id=7KAEK5CWYZQ5N4TWTYPEDLFQ7BJWJZ4V56FTMWQI2WK4IER3C75A, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:57.084335426 (60h55m6.806898476s) [noise=true , address=64.49.13.138:4483 , piece_id=6OAHEZGK7CPOQLBXETQYHPO5DW74HLCL535JARLN6KFYYZGA5P6A, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:30:57.113764934 (60h55m6.836327974s) [noise=true , address=77.90.31.13:4634 , piece_id=X6DJQMEQJ4PMFRKXZG2QTFDJX7XHWEAGMCR7YARIMNMG4FT5DJ7Q, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:31:02.476289774 (60h55m12.198852814s) [noise=true , address=77.90.31.12:4599 , piece_id=SOX3KNYYTNRJUVT7WNLRXU3JTDN2GGDJQFA3S2WU55Q6YZ452XZA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: <nil> Jul 27 00:31:46.534154650 (60h55m56.25671769s) [noise=true , address=104.36.82.53:4135 , piece_id=MBVOXZ2YFD4UFXSQJEDN6L745ET7HLK6ILOBN2I3EGZC6UNUZZYA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 188.166.225.170:56332->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:46.534197442 (60h55m56.256760472s) [noise=true , address=104.36.82.55:4195 , piece_id=VXL3FSQ3B5ILDC24DIXJSQH3GNUHL5WN2KONCPB3WEBNC4GPBJSA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: write tcp 188.166.225.170:46624->104.36.82.55:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:46.534246256 (60h55m56.256809296s) [noise=true , address=104.36.82.12:4153 , piece_id=23KWNTGZFLYY7TD35DUKAQ6EB7CZM4Z6PER7BCEEUOD2N5WN7TWA, seg_pos={0 0}, upload=58409]: Done uploading piece. err: ecclient: upload failed (node:124myE1wFhvjA5paJy5tr2hRe8jZCoEdbJLdBwvss1ZnT4GvtZv, address:104.36.82.12:4153): protocol: write tcp 188.166.225.170:34038->104.36.82.12:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:46.534270171 (60h55m56.256833211s) [noise=true , address=104.36.82.47:4528 , piece_id=5EWLBC4WA5G4JMVFOBUKUSVJAUCOP5TCYQB6WVMARPLCR7IDCPGQ, seg_pos={0 0}, upload=58409]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 188.166.225.170:46414->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:46.534295639 (60h55m56.25685868s) [noise=true , address=104.36.82.46:4487 , piece_id=V4DJD45D6XKXJACFI2BX5BU6PAOZTQGBW36SFT7KK5IT4ZSVUY5A, seg_pos={0 0}, upload=58409]: Done uploading piece. err: ecclient: upload failed (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX, address:104.36.82.46:4487): protocol: write tcp 188.166.225.170:38874->104.36.82.46:4487: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:46.534311450 (60h55m56.25687449s) [seg_pos={0 0}, upload=58409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:31:46.760925341 (60h55m56.483488391s) [upload=58409]: Done uploading segments... Jul 27 00:31:46.760954176 (60h55m56.483517206s) [upload=58409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.228499719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXX] duration: 30.236563709s FAIL (2025-07-27 00:31:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== S2EUDUYAHUXLLEUVJ3O253TXXEQZMA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:52.478235062 (60h55m2.200798102s) [seg_pos={0 0}, upload=58402]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:30:52.706576448 (60h55m2.429139479s) [upload=58402]: Done uploading segments... Jul 27 00:30:52.706600865 (60h55m2.429163905s) [upload=58402]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:31:08.011015680 (60h55m17.73357872s) [upload=58414]: Starting upload Jul 27 00:31:08.011070996 (60h55m17.733634036s) [upload=58414]: Done starting upload Jul 27 00:31:08.011140339 (60h55m17.733703369s) [upload=58414]: Uploading segments... Jul 27 00:31:08.011245279 (60h55m17.733808319s) [upload=58414]: Got next segment. Inline: false Jul 27 00:31:08.011268042 (60h55m17.733831083s) [upload=58414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:08.011272381 (60h55m17.733835411s) [upload=58414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:08.246734620 (60h55m17.96929766s) [seg_pos={0 0}, upload=58414]: Begin upload segment... Jul 27 00:31:08.246768446 (60h55m17.969331486s) [seg_pos={0 0}, upload=58414]: Done begin upload segment. Jul 27 00:31:08.246787571 (60h55m17.969350612s) [noise=true , address=45.91.116.5:4298 , piece_id=AU4QSES57R2B6FY2YR3XG4AGATE67OMNIZ76AWEHYNP6J6YCU5CQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.246821306 (60h55m17.969384346s) [noise=true , address=104.36.82.5:4837 , piece_id=JC2Q3DZYBN6X6SGQXW5HOJ2LIQB2J6TRXTJEOTYRTS67UIRAD5EA, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247118875 (60h55m17.969681915s) [noise=true , address=45.91.116.12:4227 , piece_id=FL4Y3PJNPR7OYOEXDCEORGXE6KSKE3WUVX54LZA5RXK3SUMYVZEQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247134906 (60h55m17.969697946s) [noise=true , address=104.36.82.15:4259 , piece_id=LQEHN3RWIMKIFFK7KP2OEJPEWKXQWFQ36HAL2WBD2K7JIOU36EGA, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247358033 (60h55m17.969921073s) [noise=true , address=45.91.116.14:4189 , piece_id=UTU6B2TZ5MYFQZ6CMLZNC5CHD5B2VWQP4FZTZUQNSAE2VAFI7V3Q, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247377551 (60h55m17.969940591s) [noise=true , address=104.36.82.16:4293 , piece_id=NFCIVTQYVNJJ4N3PMQA37ZSZ6IAMA7CB2DWQ37FGIOZAKA7OE3QA, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247624774 (60h55m17.970187814s) [noise=true , address=45.91.116.13:4153 , piece_id=FLSJKQ665BI6E4JMY333UWQ7HWTSKPYGD3E4Q443U6KK2SLUBBNA, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247679338 (60h55m17.970242378s) [noise=true , address=45.91.116.9:4261 , piece_id=EBW5S2SYLG2CO7CRD7XB3JB65VSM425667DLHROSMG7IAW3FRKHQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.247995573 (60h55m17.970558613s) [noise=true , address=45.91.116.4:4939 , piece_id=FT7UGC52GQFHK3HERT6ZXTJYQH7XPEYGVWN34Z4S5TWBUW6ZYCAQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248061109 (60h55m17.970624139s) [noise=true , address=45.91.116.10:4081 , piece_id=2KPZ54FDXKXVWJA3ACSZ2A2LGIODQP7AAG4W7STCBIEJS2IKHPCA, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248267223 (60h55m17.970830263s) [noise=true , address=104.36.82.6:4875 , piece_id=UNO44YFT6JQDXB2W5JULL56SS35KHD7JPMAHGZVE6W53FZAOODPQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248318070 (60h55m17.9708811s) [noise=true , address=104.36.82.4:4804 , piece_id=PUDAN444MLP6ECAY4UIFJPUWBONRX2E2KYKANDLN26EPRYJZPA6Q, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248453479 (60h55m17.971016519s) [noise=true , address=45.91.116.7:4009 , piece_id=UP4F3BKX4RX7AZKLSFINKWR6OYML7C2DB5WQ3VRRU6Z2WPOKPAAQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248505489 (60h55m17.971068529s) [noise=true , address=45.91.116.8:4045 , piece_id=BSN63GALKQMINNA65ZSM2GHOPNDBOPTLFF5I2ETUHMUNZVRDCATQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248708848 (60h55m17.971271888s) [noise=true , address=45.91.116.13:4155 , piece_id=FTL4U54L63M5S55RR6E5R5CT52TCI4CD5VDNHQ7ESDEE6YXUM7IA, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:08.248898841 (60h55m17.971461881s) [noise=true , address=45.91.116.6:4975 , piece_id=URUTBCJVUNJ43DBVITVHZZQGYPP4JJSMLOQQAKA3X43PU5ZQXEHQ, seg_pos={0 0}, upload=58414]: Uploading piece... Jul 27 00:31:09.891973506 (60h55m19.614536547s) [upload=58414]: Next returned nil segment Jul 27 00:31:09.891988885 (60h55m19.614551915s) [upload=58414]: Waiting for error group managing segments... Jul 27 00:31:10.265995744 (60h55m19.988568412s) [noise=true , address=45.91.116.14:4189 , piece_id=UTU6B2TZ5MYFQZ6CMLZNC5CHD5B2VWQP4FZTZUQNSAE2VAFI7V3Q, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.267278688 (60h55m19.989841728s) [noise=true , address=45.91.116.10:4081 , piece_id=2KPZ54FDXKXVWJA3ACSZ2A2LGIODQP7AAG4W7STCBIEJS2IKHPCA, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.344106622 (60h55m20.066669662s) [noise=true , address=45.91.116.13:4155 , piece_id=FTL4U54L63M5S55RR6E5R5CT52TCI4CD5VDNHQ7ESDEE6YXUM7IA, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.350157259 (60h55m20.072720299s) [noise=true , address=45.91.116.9:4261 , piece_id=EBW5S2SYLG2CO7CRD7XB3JB65VSM425667DLHROSMG7IAW3FRKHQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.351432359 (60h55m20.073995389s) [noise=true , address=45.91.116.13:4153 , piece_id=FLSJKQ665BI6E4JMY333UWQ7HWTSKPYGD3E4Q443U6KK2SLUBBNA, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.352131146 (60h55m20.074694186s) [noise=true , address=45.91.116.4:4939 , piece_id=FT7UGC52GQFHK3HERT6ZXTJYQH7XPEYGVWN34Z4S5TWBUW6ZYCAQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.357285969 (60h55m20.079848999s) [noise=true , address=45.91.116.12:4227 , piece_id=FL4Y3PJNPR7OYOEXDCEORGXE6KSKE3WUVX54LZA5RXK3SUMYVZEQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.366043474 (60h55m20.088606514s) [noise=true , address=45.91.116.7:4009 , piece_id=UP4F3BKX4RX7AZKLSFINKWR6OYML7C2DB5WQ3VRRU6Z2WPOKPAAQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.370830935 (60h55m20.093393975s) [noise=true , address=45.91.116.5:4298 , piece_id=AU4QSES57R2B6FY2YR3XG4AGATE67OMNIZ76AWEHYNP6J6YCU5CQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.461996031 (60h55m20.184559071s) [noise=true , address=45.91.116.8:4045 , piece_id=BSN63GALKQMINNA65ZSM2GHOPNDBOPTLFF5I2ETUHMUNZVRDCATQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:10.496427439 (60h55m20.218990489s) [noise=true , address=45.91.116.6:4975 , piece_id=URUTBCJVUNJ43DBVITVHZZQGYPP4JJSMLOQQAKA3X43PU5ZQXEHQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: <nil> Jul 27 00:31:37.372037435 (60h55m47.094600475s) [noise=true , address=104.36.82.16:4293 , piece_id=NFCIVTQYVNJJ4N3PMQA37ZSZ6IAMA7CB2DWQ37FGIOZAKA7OE3QA, seg_pos={0 0}, upload=58414]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:37.372087049 (60h55m47.094650079s) [noise=true , address=104.36.82.4:4804 , piece_id=PUDAN444MLP6ECAY4UIFJPUWBONRX2E2KYKANDLN26EPRYJZPA6Q, seg_pos={0 0}, upload=58414]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:37.372123860 (60h55m47.0946869s) [noise=true , address=104.36.82.5:4837 , piece_id=JC2Q3DZYBN6X6SGQXW5HOJ2LIQB2J6TRXTJEOTYRTS67UIRAD5EA, seg_pos={0 0}, upload=58414]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:37.372156122 (60h55m47.094719162s) [noise=true , address=104.36.82.15:4259 , piece_id=LQEHN3RWIMKIFFK7KP2OEJPEWKXQWFQ36HAL2WBD2K7JIOU36EGA, seg_pos={0 0}, upload=58414]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:37.372189485 (60h55m47.094752536s) [noise=true , address=104.36.82.6:4875 , piece_id=UNO44YFT6JQDXB2W5JULL56SS35KHD7JPMAHGZVE6W53FZAOODPQ, seg_pos={0 0}, upload=58414]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:36286->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:37.372203292 (60h55m47.094766332s) [seg_pos={0 0}, upload=58414]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:31:37.606197684 (60h55m47.328760724s) [upload=58414]: Done uploading segments... Jul 27 00:31:37.606223393 (60h55m47.328786433s) [upload=58414]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.23658491s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..X] duration: 30.001000517s FAIL (2025-07-27 00:31:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001019565s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................................X........] duration: 1.845353626s FAIL (2025-07-27 00:31:18 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.845388173s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....X...XX..XX.X.XX...X...............XX......X...XXX..XXX.] duration: 35.163043066s FAIL (2025-07-27 00:31:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.163066121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXX] duration: 30.230634338s FAIL (2025-07-27 00:30:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Z9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:42362->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013251894 (60h54m31.735814944s) [seg_pos={0 0}, upload=58394]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:30:22.242105354 (60h54m31.964668394s) [upload=58394]: Done uploading segments... Jul 27 00:30:22.242133237 (60h54m31.964696277s) [upload=58394]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:30:23.126299258 (60h54m32.848862298s) [upload=58402]: Starting upload Jul 27 00:30:23.126361067 (60h54m32.848924117s) [upload=58402]: Done starting upload Jul 27 00:30:23.126558514 (60h54m32.849121554s) [upload=58402]: Uploading segments... Jul 27 00:30:23.126592339 (60h54m32.849155369s) [upload=58402]: Got next segment. Inline: false Jul 27 00:30:23.126615263 (60h54m32.849178303s) [upload=58402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:30:23.126619842 (60h54m32.849182882s) [upload=58402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:30:23.574170560 (60h54m33.296733601s) [seg_pos={0 0}, upload=58402]: Begin upload segment... Jul 27 00:30:23.574230645 (60h54m33.296793685s) [seg_pos={0 0}, upload=58402]: Done begin upload segment. Jul 27 00:30:23.574253138 (60h54m33.296816189s) [noise=true , address=104.36.82.10:4077 , piece_id=F4GWXJQGVG33US2UUUZA5MS2EUDUYAHUXLLEUVJ3O253TXXEQZMA, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.574299768 (60h54m33.296862808s) [noise=true , address=104.36.82.2:4729 , piece_id=ZJEMWD54JHJ5KXUIJE6BYFHVJAB5SUQH6VUSG2QMZDOF5CMW2QUA, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.574808280 (60h54m33.29737132s) [noise=true , address=77.90.31.18:4805 , piece_id=VLMX6KR3DKJLGN6SRUN6POYQYEL23BZNWJ7MUB46RLE22NONHH7A, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.574905186 (60h54m33.297468226s) [noise=true , address=45.91.116.6:4973 , piece_id=J47VOT42GIMNCRFKMYRYHYSUFQBLJ4CYXA52OI2KETNFGBCBJQXQ, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.575121911 (60h54m33.297684951s) [noise=true , address=45.91.116.10:4082 , piece_id=B2IOZQMM7DWM5LZQXULKZ6UWL2ENBTVN3LS7MMGY6PJYSLWQE5WQ, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.575189340 (60h54m33.29775237s) [noise=true , address=104.36.82.14:4221 , piece_id=XQG47GCLEYV3ODAEXQQWFYZAYCKJHG4UW3SRNH6QB67SGSCMOH3Q, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.575431684 (60h54m33.297994714s) [noise=true , address=45.91.116.9:4261 , piece_id=VKTW7BSDH32O6QEHU6DI2V4EC7F4RDC3CI5C6BOM7MSTCJ5YOGQQ, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.575723252 (60h54m33.298286292s) [noise=true , address=45.91.116.14:4191 , piece_id=SEUCAVGAOA6LEV65WHDGP2C3RAOA6M3UBRFT3TCBZXX67DCHNRCQ, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.576076528 (60h54m33.298639568s) [noise=true , address=104.36.82.15:4258 , piece_id=3Q63EMFLOQHCTNUXSDVJ6DP2CTOVZIKAMDVIE44P7GZD534L5OXA, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.576448208 (60h54m33.299011249s) [noise=true , address=45.91.116.5:4297 , piece_id=5BLNTUCHQRURR2RRKDWKLUKKYLX7XRQKAWFZLIYMSCHVJGS7E26Q, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.576764614 (60h54m33.299327654s) [noise=true , address=77.90.31.12:4592 , piece_id=AOVDYG3EHFCM47IKQIUQ5O5JMR4CYUTDE3VKYYTVTLXMFDCOWERQ, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.577065019 (60h54m33.299628049s) [noise=true , address=77.90.31.11:4553 , piece_id=M46L7USXIJG7O3ZIC6GNFFG7GKVVG6HF6PNCZYWINSIZQAHFFC6A, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.577325197 (60h54m33.299888227s) [noise=true , address=45.91.116.4:4940 , piece_id=M6HWFZOQN54KW2COU6COM5WLM4LYQTDQMCY767AUIJGZDFIEYYUA, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.577559585 (60h54m33.300122626s) [noise=true , address=45.91.116.7:4011 , piece_id=2EQBGGRX24CHYONTESRBKFGD2XO2F73QDMRB3B7ZOAJ5XWK4LYMQ, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.577834281 (60h54m33.300397321s) [noise=true , address=45.91.116.11:4118 , piece_id=Z7Z6KMBEUYVTJP34DS7HKRF4FKWWXI47ZQMY2QUC5JC35WEDFNGA, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:23.578076525 (60h54m33.300639565s) [noise=true , address=45.91.116.8:4047 , piece_id=TCIGMATSUUHO76IACGGKRBIWCVGI4443IZKID3IHOIOKOE4ZTRKA, seg_pos={0 0}, upload=58402]: Uploading piece... Jul 27 00:30:25.310243360 (60h54m35.0328064s) [upload=58402]: Next returned nil segment Jul 27 00:30:25.310259861 (60h54m35.032822891s) [upload=58402]: Waiting for error group managing segments... Jul 27 00:30:25.586098263 (60h54m35.308661293s) [noise=true , address=45.91.116.8:4047 , piece_id=TCIGMATSUUHO76IACGGKRBIWCVGI4443IZKID3IHOIOKOE4ZTRKA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.590735435 (60h54m35.313298475s) [noise=true , address=45.91.116.7:4011 , piece_id=2EQBGGRX24CHYONTESRBKFGD2XO2F73QDMRB3B7ZOAJ5XWK4LYMQ, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.610687821 (60h54m35.333250871s) [noise=true , address=45.91.116.9:4261 , piece_id=VKTW7BSDH32O6QEHU6DI2V4EC7F4RDC3CI5C6BOM7MSTCJ5YOGQQ, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.610923140 (60h54m35.33348618s) [noise=true , address=45.91.116.6:4973 , piece_id=J47VOT42GIMNCRFKMYRYHYSUFQBLJ4CYXA52OI2KETNFGBCBJQXQ, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.611448064 (60h54m35.334011104s) [noise=true , address=45.91.116.10:4082 , piece_id=B2IOZQMM7DWM5LZQXULKZ6UWL2ENBTVN3LS7MMGY6PJYSLWQE5WQ, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.685279867 (60h54m35.407842907s) [noise=true , address=45.91.116.5:4297 , piece_id=5BLNTUCHQRURR2RRKDWKLUKKYLX7XRQKAWFZLIYMSCHVJGS7E26Q, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.688891708 (60h54m35.411454748s) [noise=true , address=45.91.116.4:4940 , piece_id=M6HWFZOQN54KW2COU6COM5WLM4LYQTDQMCY767AUIJGZDFIEYYUA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.753129584 (60h54m35.475692624s) [noise=true , address=45.91.116.11:4118 , piece_id=Z7Z6KMBEUYVTJP34DS7HKRF4FKWWXI47ZQMY2QUC5JC35WEDFNGA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:25.767517201 (60h54m35.490080242s) [noise=true , address=45.91.116.14:4191 , piece_id=SEUCAVGAOA6LEV65WHDGP2C3RAOA6M3UBRFT3TCBZXX67DCHNRCQ, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:27.166238117 (60h54m36.888801157s) [noise=true , address=77.90.31.18:4805 , piece_id=VLMX6KR3DKJLGN6SRUN6POYQYEL23BZNWJ7MUB46RLE22NONHH7A, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:27.816348764 (60h54m37.538911804s) [noise=true , address=77.90.31.12:4592 , piece_id=AOVDYG3EHFCM47IKQIUQ5O5JMR4CYUTDE3VKYYTVTLXMFDCOWERQ, seg_pos={0 0}, upload=58402]: Done uploading piece. err: <nil> Jul 27 00:30:52.477985996 (60h55m2.200549036s) [noise=true , address=104.36.82.14:4221 , piece_id=XQG47GCLEYV3ODAEXQQWFYZAYCKJHG4UW3SRNH6QB67SGSCMOH3Q, seg_pos={0 0}, upload=58402]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:55182->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:52.478043386 (60h55m2.200606426s) [noise=true , address=104.36.82.15:4258 , piece_id=3Q63EMFLOQHCTNUXSDVJ6DP2CTOVZIKAMDVIE44P7GZD534L5OXA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:39028->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:52.478089154 (60h55m2.200652194s) [noise=true , address=104.36.82.2:4729 , piece_id=ZJEMWD54JHJ5KXUIJE6BYFHVJAB5SUQH6VUSG2QMZDOF5CMW2QUA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:32884->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:52.478122527 (60h55m2.200685567s) [noise=true , address=77.90.31.11:4553 , piece_id=M46L7USXIJG7O3ZIC6GNFFG7GKVVG6HF6PNCZYWINSIZQAHFFC6A, seg_pos={0 0}, upload=58402]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:45846->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:52.478192952 (60h55m2.200755982s) [noise=true , address=104.36.82.10:4077 , piece_id=F4GWXJQGVG33US2UUUZA5MS2EUDUYAHUXLLEUVJ3O253TXXEQZMA, seg_pos={0 0}, upload=58402]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:52.478235062 (60h55m2.200798102s) [seg_pos={0 0}, upload=58402]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:30:52.706576448 (60h55m2.429139479s) [upload=58402]: Done uploading segments... Jul 27 00:30:52.706600865 (60h55m2.429163905s) [upload=58402]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.230650349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..] duration: 5.716923341s FAIL (2025-07-27 00:30:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.716992394s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236610807s FAIL (2025-07-27 00:30:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .165453905 (60h53m56.888016945s) [noise=true , address=104.36.82.47:4545 , piece_id=VJLT5WY4XEK2DB3RDIICHCCKZXTMJTFJF35OCKZHFBX44JKVPQ2A, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.165708383 (60h53m56.888271423s) [noise=true , address=104.36.82.46:4498 , piece_id=RKMLNZKTCB2PAAHPDNOSTVTS4THTCVGPFFXNUMCRQAZ5G44HW22A, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.165996223 (60h53m56.888559263s) [noise=true , address=104.36.82.55:4193 , piece_id=542QK6GQLFCH3DWDM6T2NBKBCYBXPZTVMZPDEAVOBBFX735AXEEA, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.166311487 (60h53m56.888874537s) [noise=true , address=104.36.82.52:4114 , piece_id=BIVH62LNAC6BCKC4DVL4BKB5WIURFCIUUYLTTZI3D463EJAF2JZA, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.166594658 (60h53m56.889157698s) [noise=true , address=77.90.31.18:4812 , piece_id=ND2OA3KK2IGUKJSH5DPOKQWGFBISIS5LYLP24TVW2U65OTWCLY2Q, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.166874353 (60h53m56.889437403s) [noise=true , address=77.90.31.15:4731 , piece_id=M2NUTMU4K7BMVFMZZRII7BVC4HUTWLZ2KERGLW4SDBFGG7OOMA5Q, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.167152896 (60h53m56.889715936s) [noise=true , address=156.146.43.225:4449 , piece_id=FAFUUDSS4ROJOPD4KC5XBP3DMZUD4S7TWNGQZ5PM2ADPOKXVIPAA, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.167404687 (60h53m56.889967717s) [noise=true , address=104.36.82.53:4153 , piece_id=23UBXJCLY57RIKZAUF6ARVT3ADAUKZV2IYI455DRW6ZUDBW6JKXQ, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.167603118 (60h53m56.890166158s) [noise=true , address=77.90.31.11:4558 , piece_id=UKHZVR6Z5DSQZZQO2NDDIYFCOEKW5AC37JO4A4LCIGLVIMUK7NLA, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.167947456 (60h53m56.890510496s) [noise=true , address=77.90.31.19:4847 , piece_id=WD5DGHGJATV365YF4DN4O4MXR2TMKL47FGRVNAB7LT4PK7LFOU3A, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.168346580 (60h53m56.89090962s) [noise=true , address=77.90.31.16:4767 , piece_id=TIFYQN2FQQKIDSH56B56F6BLE6RPZ6AFFE7TAA72EPG3JTZVPV2A, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.168359334 (60h53m56.890922374s) [noise=true , address=64.49.13.138:4488 , piece_id=2I5XTMTMJXGWQXUQIEKQIMMBGIWPR4BHF7XI3PQPLY4GUNEXZL2A, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.168628369 (60h53m56.891191409s) [noise=true , address=104.36.82.56:4258 , piece_id=MXBN7IKINLRETIBFJSS5XLDFZXYEQ3FGAGYM7RCT3PMY37CID3DQ, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.168882595 (60h53m56.891445635s) [noise=true , address=79.127.205.245:4242 , piece_id=SIJPQKWJPW5VBRDHZNKS4LGLUWUFPN5K32FGKRG47EFNLV4RUL6Q, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:47.169699869 (60h53m56.892262909s) [noise=true , address=79.127.219.38:4009 , piece_id=35MCJ3ANIXMLQRYEQSDTYPE7REGSBWC3M46M2B5AEB3RPMBG6HNA, seg_pos={0 0}, upload=58393]: Uploading piece... Jul 27 00:29:48.124921036 (60h53m57.847484076s) [noise=true , address=64.49.13.138:4488 , piece_id=2I5XTMTMJXGWQXUQIEKQIMMBGIWPR4BHF7XI3PQPLY4GUNEXZL2A, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb, address:64.49.13.138:4488): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:48.192417837 (60h53m57.914980877s) [noise=true , address=77.90.31.15:4731 , piece_id=M2NUTMU4K7BMVFMZZRII7BVC4HUTWLZ2KERGLW4SDBFGG7OOMA5Q, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, address:77.90.31.15:4731): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:48.285263929 (60h53m58.007826959s) [upload=58393]: Next returned nil segment Jul 27 00:29:48.285316941 (60h53m58.007879991s) [upload=58393]: Waiting for error group managing segments... Jul 27 00:29:48.588696957 (60h53m58.311259997s) [noise=true , address=79.127.205.245:4242 , piece_id=SIJPQKWJPW5VBRDHZNKS4LGLUWUFPN5K32FGKRG47EFNLV4RUL6Q, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:29:49.181671547 (60h53m58.904234587s) [noise=true , address=79.127.219.38:4009 , piece_id=35MCJ3ANIXMLQRYEQSDTYPE7REGSBWC3M46M2B5AEB3RPMBG6HNA, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:29:49.210624713 (60h53m58.933187753s) [noise=true , address=45.91.116.10:4093 , piece_id=RMS575PC4V5HQSHMZBBVD7CCPDFCJ7RQ7B4ACXD4BMYLB76QJAQQ, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:29:49.271256259 (60h53m58.993819299s) [noise=true , address=77.90.31.16:4767 , piece_id=TIFYQN2FQQKIDSH56B56F6BLE6RPZ6AFFE7TAA72EPG3JTZVPV2A, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:29:49.956730909 (60h53m59.679293949s) [noise=true , address=156.146.43.225:4449 , piece_id=FAFUUDSS4ROJOPD4KC5XBP3DMZUD4S7TWNGQZ5PM2ADPOKXVIPAA, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:29:59.946028360 (60h54m9.6685914s) [noise=true , address=77.90.31.19:4847 , piece_id=WD5DGHGJATV365YF4DN4O4MXR2TMKL47FGRVNAB7LT4PK7LFOU3A, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:30:14.665507580 (60h54m24.38807062s) [noise=true , address=77.90.31.18:4812 , piece_id=ND2OA3KK2IGUKJSH5DPOKQWGFBISIS5LYLP24TVW2U65OTWCLY2Q, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:30:26.994662169 (60h54m36.717225209s) [noise=true , address=77.90.31.13:4643 , piece_id=RA4EAYTCNURHPGS3E3RZFE5FYVDPHZS4HJ2XZPWNLBAZQNUOJ2ZA, seg_pos={0 0}, upload=58393]: Done uploading piece. err: <nil> Jul 27 00:30:46.065432435 (60h54m55.787995475s) [noise=true , address=104.36.82.52:4114 , piece_id=BIVH62LNAC6BCKC4DVL4BKB5WIURFCIUUYLTTZI3D463EJAF2JZA, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: write tcp 188.166.225.170:59500->104.36.82.52:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065475047 (60h54m55.788038087s) [noise=true , address=104.36.82.46:4498 , piece_id=RKMLNZKTCB2PAAHPDNOSTVTS4THTCVGPFFXNUMCRQAZ5G44HW22A, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: write tcp 188.166.225.170:34392->104.36.82.46:4498: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065781203 (60h54m55.788344243s) [noise=true , address=77.90.31.11:4558 , piece_id=UKHZVR6Z5DSQZZQO2NDDIYFCOEKW5AC37JO4A4LCIGLVIMUK7NLA, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:37630->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065840927 (60h54m55.788403967s) [noise=true , address=104.36.82.11:4143 , piece_id=U2DXIIE5ZZAYJTCX26WP65JYJZFD5DYUM4VGTWQPQ5MQDMI3CL5Q, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: write tcp 188.166.225.170:57238->104.36.82.11:4143: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065871726 (60h54m55.788434756s) [noise=true , address=104.36.82.55:4193 , piece_id=542QK6GQLFCH3DWDM6T2NBKBCYBXPZTVMZPDEAVOBBFX735AXEEA, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: write tcp 188.166.225.170:55110->104.36.82.55:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065876234 (60h54m55.788439274s) [noise=true , address=104.36.82.53:4153 , piece_id=23UBXJCLY57RIKZAUF6ARVT3ADAUKZV2IYI455DRW6ZUDBW6JKXQ, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:38398->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065903557 (60h54m55.788466597s) [noise=true , address=104.36.82.47:4545 , piece_id=VJLT5WY4XEK2DB3RDIICHCCKZXTMJTFJF35OCKZHFBX44JKVPQ2A, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 188.166.225.170:51066->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065920960 (60h54m55.788484s) [noise=true , address=104.36.82.56:4258 , piece_id=MXBN7IKINLRETIBFJSS5XLDFZXYEQ3FGAGYM7RCT3PMY37CID3DQ, seg_pos={0 0}, upload=58393]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:42536->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:46.065933575 (60h54m55.788496605s) [seg_pos={0 0}, upload=58393]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:30:46.300627042 (60h54m56.023190082s) [upload=58393]: Done uploading segments... Jul 27 00:30:46.300650207 (60h54m56.023213247s) [upload=58393]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236624474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................................X.......] duration: 1.84029096s FAIL (2025-07-27 00:30:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.84031715s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [......X...XX..XX.X.XX...X...............XX......X...XXX..XXX] duration: 1m0.001176066s FAIL (2025-07-27 00:30:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001190973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXX] duration: 30.230738746s FAIL (2025-07-27 00:30:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:14.180912281 (60h53m23.903475321s) [upload=58374]: Done uploading segments... Jul 27 00:29:14.180949542 (60h53m23.903512582s) [upload=58374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:29:52.655472786 (60h54m2.378035826s) [upload=58394]: Starting upload Jul 27 00:29:52.655531158 (60h54m2.378094198s) [upload=58394]: Done starting upload Jul 27 00:29:52.655627351 (60h54m2.378190381s) [upload=58394]: Uploading segments... Jul 27 00:29:52.655706563 (60h54m2.378269603s) [upload=58394]: Got next segment. Inline: false Jul 27 00:29:52.655747512 (60h54m2.378310552s) [upload=58394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:29:52.655752682 (60h54m2.378315722s) [upload=58394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:29:52.888121222 (60h54m2.610684262s) [seg_pos={0 0}, upload=58394]: Begin upload segment... Jul 27 00:29:52.888162492 (60h54m2.610725532s) [seg_pos={0 0}, upload=58394]: Done begin upload segment. Jul 27 00:29:52.888185005 (60h54m2.610748045s) [noise=true , address=45.91.116.4:4940 , piece_id=ZX2KGDIA4KKRZB7VTMZMEANSMUQK47MNXS4XRX465UZNNMZFEJXQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888194152 (60h54m2.610757192s) [noise=true , address=77.90.31.21:4879 , piece_id=VCXVA6S4AD6TZB64JMKAEKFW6AKIIQTCNPRHTU3AYNEHIQ4DHAVA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888250500 (60h54m2.61081353s) [noise=true , address=77.90.31.12:4589 , piece_id=4I7EGZ6EXMBIK6TTGBHGAI45IPRBTP67W6OT2YHN74RQFD2XPZEQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888546647 (60h54m2.611109717s) [noise=true , address=77.90.31.14:4661 , piece_id=K3SYCFI2OP6C3Z7EFVKZJFDVZ3ZQWMCHFRRUHM4EP67BD6PJ2I4Q, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888609988 (60h54m2.611173028s) [noise=true , address=104.36.82.14:4221 , piece_id=5XRWLPOJHXUWFNSHPXTOQBETN7AM24KNEONYBDUEXIQDQE2JCBMA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888820371 (60h54m2.611383411s) [noise=true , address=104.36.82.4:4801 , piece_id=ADCR6YLPHB7VXYCIXT6HLR3ACJ3FHBQLYKIPWOQWCBXPEOHQQBGA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888918027 (60h54m2.611481067s) [noise=true , address=45.91.116.6:4975 , piece_id=6FGXCSAM4SKKNPHFHF23LMAI4TDQV7N6MMZTX4PJGL5EXDFUSIDQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889061982 (60h54m2.611625022s) [noise=true , address=104.36.82.16:4293 , piece_id=AO42DKLBS6QAXDMJ4K2TKFX5RE4UTWDWKYI4B6HKYCVVB3WWD4CA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889165400 (60h54m2.61172844s) [noise=true , address=45.91.116.7:4011 , piece_id=AWQG67T4DCFF53T4D47YJ7W3N367UXZUC27K4MGLK4HTJL6KA5OA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889307111 (60h54m2.611870151s) [noise=true , address=45.91.116.11:4117 , piece_id=2626XHGTTVACC5NOUZ7NYH6FSMUVJ4B32K5XSPHQSGBQ6CWGJ57A, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889373619 (60h54m2.611936659s) [noise=true , address=77.90.31.11:4555 , piece_id=PXT2TZSUXDTKPIEWREAG224KZGJZ4333GG3PQ5V6PN3M4WTBMKBQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889567510 (60h54m2.61213054s) [noise=true , address=77.90.31.19:4843 , piece_id=LKKIBE65SCED3XZ2LKAJBJEGQGDVHFWXDAJWNSF7TSWK65SPZF5Q, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889786339 (60h54m2.612349379s) [noise=true , address=45.91.116.12:4226 , piece_id=7EB6ZQJLAHN5ZUY6V6XI2ZMSDP5A3YXJ5AWVLUC7KRA4H7EEV6FA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.890008103 (60h54m2.612571143s) [noise=true , address=45.91.116.8:4047 , piece_id=JAQBSAZQVW7MPINI4BELMFMYBWYREGR3S7EARRDVLKO3FDOMWCZA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.890207524 (60h54m2.612770554s) [noise=true , address=104.36.82.11:4115 , piece_id=ODD6JFRCK5XE3VA6N2HF6TDNIUPTZTTBDCEFWDT4UVWVN4NS5K6Q, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.890594855 (60h54m2.613157895s) [noise=true , address=77.90.31.15:4697 , piece_id=DT7CTZTYNNEJPKHK435O3XVYZNJ2TI55GJMLCM4UVXN5IBYBZ5FA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:54.628543392 (60h54m4.351106432s) [upload=58394]: Next returned nil segment Jul 27 00:29:54.628569472 (60h54m4.351132512s) [upload=58394]: Waiting for error group managing segments... Jul 27 00:29:54.909550974 (60h54m4.632114014s) [noise=true , address=45.91.116.6:4975 , piece_id=6FGXCSAM4SKKNPHFHF23LMAI4TDQV7N6MMZTX4PJGL5EXDFUSIDQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:54.915602013 (60h54m4.638165043s) [noise=true , address=45.91.116.8:4047 , piece_id=JAQBSAZQVW7MPINI4BELMFMYBWYREGR3S7EARRDVLKO3FDOMWCZA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:54.995497366 (60h54m4.718060406s) [noise=true , address=45.91.116.11:4117 , piece_id=2626XHGTTVACC5NOUZ7NYH6FSMUVJ4B32K5XSPHQSGBQ6CWGJ57A, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:54.996183028 (60h54m4.718746068s) [noise=true , address=77.90.31.21:4879 , piece_id=VCXVA6S4AD6TZB64JMKAEKFW6AKIIQTCNPRHTU3AYNEHIQ4DHAVA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.013792189 (60h54m4.736355229s) [noise=true , address=45.91.116.7:4011 , piece_id=AWQG67T4DCFF53T4D47YJ7W3N367UXZUC27K4MGLK4HTJL6KA5OA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.053289159 (60h54m4.775852189s) [noise=true , address=77.90.31.14:4661 , piece_id=K3SYCFI2OP6C3Z7EFVKZJFDVZ3ZQWMCHFRRUHM4EP67BD6PJ2I4Q, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.066042380 (60h54m4.78860542s) [noise=true , address=45.91.116.4:4940 , piece_id=ZX2KGDIA4KKRZB7VTMZMEANSMUQK47MNXS4XRX465UZNNMZFEJXQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.066112193 (60h54m4.788675223s) [noise=true , address=45.91.116.12:4226 , piece_id=7EB6ZQJLAHN5ZUY6V6XI2ZMSDP5A3YXJ5AWVLUC7KRA4H7EEV6FA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:56.415866718 (60h54m6.138429759s) [noise=true , address=77.90.31.12:4589 , piece_id=4I7EGZ6EXMBIK6TTGBHGAI45IPRBTP67W6OT2YHN74RQFD2XPZEQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:56.421881577 (60h54m6.144444677s) [noise=true , address=77.90.31.19:4843 , piece_id=LKKIBE65SCED3XZ2LKAJBJEGQGDVHFWXDAJWNSF7TSWK65SPZF5Q, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:30:22.013004291 (60h54m31.735567331s) [noise=true , address=104.36.82.14:4221 , piece_id=5XRWLPOJHXUWFNSHPXTOQBETN7AM24KNEONYBDUEXIQDQE2JCBMA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013049246 (60h54m31.735612286s) [noise=true , address=104.36.82.11:4115 , piece_id=ODD6JFRCK5XE3VA6N2HF6TDNIUPTZTTBDCEFWDT4UVWVN4NS5K6Q, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013095164 (60h54m31.735658214s) [noise=true , address=77.90.31.15:4697 , piece_id=DT7CTZTYNNEJPKHK435O3XVYZNJ2TI55GJMLCM4UVXN5IBYBZ5FA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:33604->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:33604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33604->77.90.31.15:4697: use of closed network connection Jul 27 00:30:22.013122086 (60h54m31.735685136s) [noise=true , address=104.36.82.4:4801 , piece_id=ADCR6YLPHB7VXYCIXT6HLR3ACJ3FHBQLYKIPWOQWCBXPEOHQQBGA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:46330->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013143317 (60h54m31.735706387s) [noise=true , address=77.90.31.11:4555 , piece_id=PXT2TZSUXDTKPIEWREAG224KZGJZ4333GG3PQ5V6PN3M4WTBMKBQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013204123 (60h54m31.735767193s) [noise=true , address=104.36.82.16:4293 , piece_id=AO42DKLBS6QAXDMJ4K2TKFX5RE4UTWDWKYI4B6HKYCVVB3WWD4CA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:42362->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013251894 (60h54m31.735814944s) [seg_pos={0 0}, upload=58394]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:30:22.242105354 (60h54m31.964668394s) [upload=58394]: Done uploading segments... Jul 27 00:30:22.242133237 (60h54m31.964696277s) [upload=58394]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.230826955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX.] duration: 7.645276494s FAIL (2025-07-27 00:30:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 7.645302825s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..] duration: 7.64924155s FAIL (2025-07-27 00:30:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.649258583s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235400757s FAIL (2025-07-27 00:29:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 8377]: Uploading piece... Jul 27 00:28:46.711272766 (60h52m56.433835806s) [noise=true , address=77.90.31.15:4721 , piece_id=HPMTWFFN6P7ZVSY7YC3QZSZ262BG34RR7QHLGRMTDS73UIMVZRBQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.711650058 (60h52m56.434213098s) [noise=true , address=79.127.201.215:4343 , piece_id=67FFLJWPZS67MRE32UIZ57AQXTDSUXNJ2W2T7HWNAQC44YZQG7OA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.711999758 (60h52m56.434562798s) [noise=true , address=104.36.82.14:4253 , piece_id=NHGD2RHKC57F53LMVMLZ7WBJ6ZRRSMZTLFNYBT4TKKQXZSC2TC7Q, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.712303679 (60h52m56.434866719s) [noise=true , address=79.127.205.245:4250 , piece_id=KLHAXR77NJLF4Y3TZJH23BEFZOKTK5FOFLXJNTWF4RZDWQJDX4HA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.712564868 (60h52m56.435127908s) [noise=true , address=77.90.31.17:4780 , piece_id=UGYIRQSDALEK6UHYB5AWHQ5OE67LDQIVYWBFRLRQMH6FWRD5S5EQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.712795831 (60h52m56.435358871s) [noise=true , address=104.36.82.53:4128 , piece_id=7BPHJUQKVUPTFEWGLR4M3SXLBMZ7AI6PZK3D73XRFJHSVR64PKMQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713093891 (60h52m56.435656931s) [noise=true , address=79.127.205.233:4640 , piece_id=3OSVXOSUOFWHBYAFAADMYB5I5BJYZMEPI7L27L2WMFSC425DKOHA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713347416 (60h52m56.435910456s) [noise=true , address=45.91.116.9:4277 , piece_id=PMOIKM7MMDDZWPORYSCBB74QWP2RP7T4RD6M5QD7KLLE563AVCTQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713594198 (60h52m56.436157238s) [noise=true , address=104.36.82.52:4108 , piece_id=CVBA7Z5FR3GIANFC45ZWPRBY4Z7PIE7O3PP4UAGCKS4BQ6V4MHNA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713864205 (60h52m56.436427245s) [noise=true , address=104.36.82.55:4197 , piece_id=PMZZPS6CJ6GCGFUNXHYYE3SEZ3S7JO4E6PT7MQWG7T6C56H576SA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.714178506 (60h52m56.436741536s) [noise=true , address=104.36.82.46:4503 , piece_id=LHAAZC3BEQR4UWUF2ZROVP36PUU73DBS257DQMRLYSFWX62MWVAA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.714431810 (60h52m56.43699485s) [noise=true , address=77.90.31.16:4743 , piece_id=ZFQRHC7UVDI2RSWJTMKQ5XNPDKHS3YS35DZPI53WUQMEE3QJHQHA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.714670818 (60h52m56.437233858s) [noise=true , address=64.49.13.138:4482 , piece_id=TL7QXM5NTLREIZ6XZ3EFFD6M7T7BNSG4IVSQBL4IT5SLURKF6ACA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.715522918 (60h52m56.438085948s) [noise=true , address=79.127.205.235:4690 , piece_id=J5OMFH6HJGEVFXOC67ZOSVJZSBTF3BUR5R2QVZ67VYIYQ7A6SO5Q, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.715770181 (60h52m56.438333221s) [noise=true , address=79.127.219.38:4017 , piece_id=BNOC6NR3TITKYEXYZCRIRCAQL677QTNJJY4GHMZ4KFQKWDDR3JKA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:47.798958155 (60h52m57.521521185s) [noise=true , address=77.90.31.15:4721 , piece_id=HPMTWFFN6P7ZVSY7YC3QZSZ262BG34RR7QHLGRMTDS73UIMVZRBQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn, address:77.90.31.15:4721): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:47.800722480 (60h52m57.52328552s) [noise=true , address=77.90.31.12:4621 , piece_id=KIUWSLCHV5HAH5BBZ6PFCJVBFDUZM425G6FC3QA2GCOYPCRZQAAA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:47.815528690 (60h52m57.53809173s) [upload=58377]: Next returned nil segment Jul 27 00:28:47.815545141 (60h52m57.538108171s) [upload=58377]: Waiting for error group managing segments... Jul 27 00:28:48.101555706 (60h52m57.824118746s) [noise=true , address=79.127.205.234:4608 , piece_id=RSMIE75BH7MRWOYWFZZWT7UQFHTTFPULYG6MBAY3F4R4RH25BEIA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.117363039 (60h52m57.839926079s) [noise=true , address=79.127.205.235:4690 , piece_id=J5OMFH6HJGEVFXOC67ZOSVJZSBTF3BUR5R2QVZ67VYIYQ7A6SO5Q, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.129849609 (60h52m57.852412639s) [noise=true , address=79.127.205.233:4640 , piece_id=3OSVXOSUOFWHBYAFAADMYB5I5BJYZMEPI7L27L2WMFSC425DKOHA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.189088602 (60h52m57.911651632s) [noise=true , address=64.49.13.138:4482 , piece_id=TL7QXM5NTLREIZ6XZ3EFFD6M7T7BNSG4IVSQBL4IT5SLURKF6ACA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.213068062 (60h52m57.935631102s) [noise=true , address=45.91.116.9:4277 , piece_id=PMOIKM7MMDDZWPORYSCBB74QWP2RP7T4RD6M5QD7KLLE563AVCTQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:48.526260233 (60h52m58.248823273s) [noise=true , address=79.127.205.245:4250 , piece_id=KLHAXR77NJLF4Y3TZJH23BEFZOKTK5FOFLXJNTWF4RZDWQJDX4HA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.627462864 (60h52m58.350025904s) [noise=true , address=79.127.219.38:4017 , piece_id=BNOC6NR3TITKYEXYZCRIRCAQL677QTNJJY4GHMZ4KFQKWDDR3JKA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:29:06.829790417 (60h53m16.552353457s) [noise=true , address=79.127.201.215:4343 , piece_id=67FFLJWPZS67MRE32UIZ57AQXTDSUXNJ2W2T7HWNAQC44YZQG7OA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:29:45.612427409 (60h53m55.334990449s) [noise=true , address=104.36.82.14:4253 , piece_id=NHGD2RHKC57F53LMVMLZ7WBJ6ZRRSMZTLFNYBT4TKKQXZSC2TC7Q, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: rpc: dial tcp 104.36.82.14:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612532229 (60h53m55.335095269s) [noise=true , address=104.36.82.52:4108 , piece_id=CVBA7Z5FR3GIANFC45ZWPRBY4Z7PIE7O3PP4UAGCKS4BQ6V4MHNA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: rpc: dial tcp 104.36.82.52:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612567316 (60h53m55.335130346s) [noise=true , address=77.90.31.17:4780 , piece_id=UGYIRQSDALEK6UHYB5AWHQ5OE67LDQIVYWBFRLRQMH6FWRD5S5EQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: read tcp 188.166.225.170:54018->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612591222 (60h53m55.335154262s) [noise=true , address=104.36.82.53:4128 , piece_id=7BPHJUQKVUPTFEWGLR4M3SXLBMZ7AI6PZK3D73XRFJHSVR64PKMQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:44532->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612686113 (60h53m55.335249153s) [noise=true , address=104.36.82.55:4197 , piece_id=PMZZPS6CJ6GCGFUNXHYYE3SEZ3S7JO4E6PT7MQWG7T6C56H576SA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612761338 (60h53m55.335324378s) [noise=true , address=77.90.31.16:4743 , piece_id=ZFQRHC7UVDI2RSWJTMKQ5XNPDKHS3YS35DZPI53WUQMEE3QJHQHA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612782699 (60h53m55.335345739s) [noise=true , address=104.36.82.47:4540 , piece_id=DAX3KOH67IM3D346THSU5THSNMKQZQ3U4KMA7243BEPYKJGSTICA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:104.36.82.47:4540): protocol: rpc: dial tcp 104.36.82.47:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612811414 (60h53m55.335374454s) [noise=true , address=104.36.82.46:4503 , piece_id=LHAAZC3BEQR4UWUF2ZROVP36PUU73DBS257DQMRLYSFWX62MWVAA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 188.166.225.170:53506->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612831541 (60h53m55.335394581s) [seg_pos={0 0}, upload=58377]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:29:45.846130043 (60h53m55.568693083s) [upload=58377]: Done uploading segments... Jul 27 00:29:45.846154379 (60h53m55.568717409s) [upload=58377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.235469478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX] duration: 30.232921317s FAIL (2025-07-27 00:29:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.232946877s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X......X...XX..XX.X.XX...X...............XX......X...XXX..XX] duration: 1m0.000993634s FAIL (2025-07-27 00:29:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001007762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................................X......] duration: 2.017639552s FAIL (2025-07-27 00:29:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.017662425s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXX] duration: 30.233809814s FAIL (2025-07-27 00:29:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070670444 (60h52m25.793233484s) [noise=true , address=104.36.82.10:4077 , piece_id=DNXWQ2K4RPQBH7CBRGBNY3IYE34K5SOJUUR7563HC4VD4FG2LUQA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070695342 (60h52m25.793258382s) [seg_pos={0 0}, upload=58361]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:28:16.311840156 (60h52m26.034403206s) [upload=58361]: Done uploading segments... Jul 27 00:28:16.311867668 (60h52m26.034430719s) [upload=58361]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:28:44.591999571 (60h52m54.314562611s) [upload=58374]: Starting upload Jul 27 00:28:44.592052462 (60h52m54.314615502s) [upload=58374]: Done starting upload Jul 27 00:28:44.592228058 (60h52m54.314791098s) [upload=58374]: Uploading segments... Jul 27 00:28:44.592259758 (60h52m54.314822798s) [upload=58374]: Got next segment. Inline: false Jul 27 00:28:44.592281730 (60h52m54.31484477s) [upload=58374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:44.592285858 (60h52m54.314848898s) [upload=58374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:44.828464899 (60h52m54.551027939s) [seg_pos={0 0}, upload=58374]: Begin upload segment... Jul 27 00:28:44.828522289 (60h52m54.551085329s) [seg_pos={0 0}, upload=58374]: Done begin upload segment. Jul 27 00:28:44.828546966 (60h52m54.551110006s) [noise=true , address=45.91.116.11:4117 , piece_id=RKKXC72UC7PCU34L6G6BBWLJOFGQ346CNXQD7LEVX3ORVNTNLGOQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.828578948 (60h52m54.551141988s) [noise=true , address=45.91.116.13:4153 , piece_id=JFZFNR2HRKDLHRZVL2RGEIHDO764HN6L65I5CINGEJUOBMFRGNSQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.828888089 (60h52m54.551451119s) [noise=true , address=45.91.116.6:4974 , piece_id=X3BDZCDSHFBFJKLXNHHRCA3DXTLGBJL3ZTLOUZVW5HIM72LRCVDA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.829044568 (60h52m54.551607608s) [noise=true , address=45.91.116.14:4190 , piece_id=PZI3W4CV5BPSBK2N5DXX5JM6TU6KBTOEQKYZZJZSVHQX2ORLUOEQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.829151703 (60h52m54.551714733s) [noise=true , address=104.36.82.14:4221 , piece_id=E4XQ5ASV6APGJAY56SKEEQFR22BIWXZIH5ECX2RTMTZDZH6CLYHA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.829400960 (60h52m54.551964s) [noise=true , address=45.91.116.4:4937 , piece_id=BQMK2EDRO7Z26VPUVHCKIQNDSNL4RSUXJHXPIBMDMHOZLSAZYWJA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.830312715 (60h52m54.552875745s) [noise=true , address=77.90.31.12:4592 , piece_id=DREP7KTKNSSVYIBLPJ6NCY6WJ4FKJN7H3B67RVFJUA4V3K2ZT2OA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.830924305 (60h52m54.553487345s) [noise=true , address=77.90.31.15:4697 , piece_id=QQDPDWCV3O5MC7YLCCH4RC4IAGG4CZZ3WKHXTFIHMSXUCRPUZDTA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.831164435 (60h52m54.553727475s) [noise=true , address=104.36.82.15:4259 , piece_id=FJRYGEYB77KMKPYCHNWDYOJH5GXFVRKHACWCXBJTEJFGGN6Q7AVQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.831405225 (60h52m54.553968265s) [noise=true , address=104.36.82.5:4838 , piece_id=ZQJ4T6IKM7EUDFT7SMWJ2PPARIMKI6HHIN5Z3ZLBNM3LPEILR2PA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.831995495 (60h52m54.554558535s) [noise=true , address=45.91.116.5:4297 , piece_id=5OO73VRTU4GJLXEVIMHMYENQN2M5LMO6PPWIFC7I5SHTEJDKDPDA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832227989 (60h52m54.554791029s) [noise=true , address=77.90.31.10:4520 , piece_id=DR52FBH7LD4YOQVE4WXXTRHCWQQN4ZTFIL3XI7EIKA5WD254XMIQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832444824 (60h52m54.555007854s) [noise=true , address=77.90.31.14:4662 , piece_id=LTHBKUVGUD65XZZLLSJYXUSCZNCDAKZXH4I6EV5O3TIUQ26KTWNQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832647652 (60h52m54.555210682s) [noise=true , address=77.90.31.19:4843 , piece_id=C3Y46YZNOIW5LH2PCACMWDN3GIMYNDNP5OM3RHOTVMF4OOV2DMLQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832862052 (60h52m54.555425092s) [noise=true , address=45.91.116.7:4009 , piece_id=W2ABMWJM4KVWOALTXO5AASOTBFG5EEIFHJA77277HV7STXXAZ2HA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.833092333 (60h52m54.555655373s) [noise=true , address=104.36.82.16:4295 , piece_id=RRC63VBCQLLURXOSQSKN7VPFBX6MAGYVNW2L2AKREZT7QFCLYT3A, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:46.567879391 (60h52m56.290442432s) [upload=58374]: Next returned nil segment Jul 27 00:28:46.567921533 (60h52m56.290484563s) [upload=58374]: Waiting for error group managing segments... Jul 27 00:28:46.859557640 (60h52m56.58212068s) [noise=true , address=45.91.116.4:4937 , piece_id=BQMK2EDRO7Z26VPUVHCKIQNDSNL4RSUXJHXPIBMDMHOZLSAZYWJA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.925038915 (60h52m56.647601955s) [noise=true , address=45.91.116.6:4974 , piece_id=X3BDZCDSHFBFJKLXNHHRCA3DXTLGBJL3ZTLOUZVW5HIM72LRCVDA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.947189906 (60h52m56.669752946s) [noise=true , address=45.91.116.13:4153 , piece_id=JFZFNR2HRKDLHRZVL2RGEIHDO764HN6L65I5CINGEJUOBMFRGNSQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.951344034 (60h52m56.673907074s) [noise=true , address=45.91.116.14:4190 , piece_id=PZI3W4CV5BPSBK2N5DXX5JM6TU6KBTOEQKYZZJZSVHQX2ORLUOEQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.984783554 (60h52m56.707346584s) [noise=true , address=45.91.116.5:4297 , piece_id=5OO73VRTU4GJLXEVIMHMYENQN2M5LMO6PPWIFC7I5SHTEJDKDPDA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.988573646 (60h52m56.711136676s) [noise=true , address=77.90.31.14:4662 , piece_id=LTHBKUVGUD65XZZLLSJYXUSCZNCDAKZXH4I6EV5O3TIUQ26KTWNQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:47.025300185 (60h52m56.747863225s) [noise=true , address=45.91.116.7:4009 , piece_id=W2ABMWJM4KVWOALTXO5AASOTBFG5EEIFHJA77277HV7STXXAZ2HA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:47.057579054 (60h52m56.780142094s) [noise=true , address=45.91.116.11:4117 , piece_id=RKKXC72UC7PCU34L6G6BBWLJOFGQ346CNXQD7LEVX3ORVNTNLGOQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:48.284570658 (60h52m58.007133698s) [noise=true , address=77.90.31.19:4843 , piece_id=C3Y46YZNOIW5LH2PCACMWDN3GIMYNDNP5OM3RHOTVMF4OOV2DMLQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:48.975836947 (60h52m58.698399987s) [noise=true , address=77.90.31.15:4697 , piece_id=QQDPDWCV3O5MC7YLCCH4RC4IAGG4CZZ3WKHXTFIHMSXUCRPUZDTA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:50.267311737 (60h52m59.989874777s) [noise=true , address=77.90.31.12:4592 , piece_id=DREP7KTKNSSVYIBLPJ6NCY6WJ4FKJN7H3B67RVFJUA4V3K2ZT2OA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:29:06.284041646 (60h53m16.006604686s) [noise=true , address=77.90.31.10:4520 , piece_id=DR52FBH7LD4YOQVE4WXXTRHCWQQN4ZTFIL3XI7EIKA5WD254XMIQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:29:13.948811773 (60h53m23.671374813s) [noise=true , address=104.36.82.15:4259 , piece_id=FJRYGEYB77KMKPYCHNWDYOJH5GXFVRKHACWCXBJTEJFGGN6Q7AVQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948882518 (60h53m23.671445558s) [noise=true , address=104.36.82.16:4295 , piece_id=RRC63VBCQLLURXOSQSKN7VPFBX6MAGYVNW2L2AKREZT7QFCLYT3A, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948915532 (60h53m23.671478572s) [noise=true , address=104.36.82.14:4221 , piece_id=E4XQ5ASV6APGJAY56SKEEQFR22BIWXZIH5ECX2RTMTZDZH6CLYHA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948936541 (60h53m23.671499581s) [noise=true , address=104.36.82.5:4838 , piece_id=ZQJ4T6IKM7EUDFT7SMWJ2PPARIMKI6HHIN5Z3ZLBNM3LPEILR2PA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:55422->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948958303 (60h53m23.671521343s) [seg_pos={0 0}, upload=58374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:14.180912281 (60h53m23.903475321s) [upload=58374]: Done uploading segments... Jul 27 00:29:14.180949542 (60h53m23.903512582s) [upload=58374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233839369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X.] duration: 17.867544543s FAIL (2025-07-27 00:29:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.867568759s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.X] duration: 30.232811868s FAIL (2025-07-27 00:28:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.232831326s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238540304s FAIL (2025-07-27 00:28:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=77.90.31.13:4659 , piece_id=NZRBSSZVEKLDUKJB47TA77QEFZ5GVGJAGHA7CH3QCZ4KIK5IRHWQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722113659 (60h51m48.444676699s) [noise=true , address=104.36.82.54:4181 , piece_id=MOMRZY5WTQN7BA7VNTL2MMYNSLDFGKVFHYBTYONSVCJLPQB4B7CA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722161811 (60h51m48.444724851s) [noise=true , address=45.91.116.4:4964 , piece_id=HZ33J55E4NIGVAXCFEH6IYKCDVKWWAEBWPM5EQKK3L4TGEGT6G5Q, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722418303 (60h51m48.444981343s) [noise=true , address=79.127.205.246:4275 , piece_id=JHTG62SBAUGKPOG2S5Q57H5IVZSZ4AL53AZSBJM2NLF6NEVQSOTA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722427781 (60h51m48.444990821s) [noise=true , address=104.36.82.55:4210 , piece_id=EVQD3W7LZ6ZE22WYOLYWEBKSDFFLTFDTVQ4CRFCWAMA5B45AXJ2A, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722681786 (60h51m48.445244826s) [noise=true , address=104.36.82.46:4498 , piece_id=5YJIJDWY62EP5B2NUKBF2UX7J2KVGHPZOYB6WT2W6FERLNXZL3UA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722699210 (60h51m48.44526224s) [noise=true , address=64.49.13.26:4953 , piece_id=H37PPSMDWBKRF4LUKWBAGZRQG4423L2GQTGXOPUU323RRACJHLQA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722975438 (60h51m48.445538468s) [noise=true , address=104.36.82.52:4092 , piece_id=HPHY4OA25YF5DA55RTMOIH2V33BXQ34ORVOE5Z7C4J2KCEVNVPXA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722982141 (60h51m48.445545181s) [noise=true , address=79.127.205.235:4668 , piece_id=6F5PMSOXC6DK2YLFXJFYPRI4NFTUYSGDISXUX5MBA4L7XHOTU6OA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.723008101 (60h51m48.445571141s) [noise=true , address=77.90.31.21:4905 , piece_id=CJOB6OCGNGQXPPXELLMKP22LMZJIJZAJ3U4FZBI7526CMAH7FABA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.723294879 (60h51m48.445857909s) [noise=true , address=104.36.82.47:4518 , piece_id=OFRKVCFF7I42OTYMUGXMPQJKTMJWX2XBQ2TWEJIRIPP7H3VPJVBA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.724096013 (60h51m48.446659053s) [noise=true , address=77.90.31.16:4761 , piece_id=SSQPNG7UZS6GDFPWQLHTUIXBN44ZBBMN2QIGXEXYRVTFLGGMEAKA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.724377000 (60h51m48.44694004s) [noise=true , address=79.127.219.38:4005 , piece_id=DR5TRODJ5CVJ4SQCEHICJP3UMDZVDMIHSNSRFEXJZAXZA6CNOP2A, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.724971457 (60h51m48.447534487s) [noise=true , address=79.127.205.234:4616 , piece_id=FGIQTFKQHUEHCHBJ2N6BRGXFOI36QO46VKL3QIUDUVMMVRJ5ZTTQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.725395689 (60h51m48.447958729s) [noise=true , address=64.49.13.138:4488 , piece_id=FPCV3SLHC5KPBO5LIUB4WTKE3HIK6SVPFLVSB7C5ZC4I22FDALUQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.729229584 (60h51m48.451792614s) [noise=true , address=104.36.82.56:4235 , piece_id=Z4F3TG2WMMHBCDHNZOGZQP6UT25R4NQ6ZPZQWCLMKN4TFOXHQYEA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.731118058 (60h51m48.453681098s) [noise=true , address=104.36.82.5:4866 , piece_id=RR6542LBU2TDSMU7UKHK45HYA4R2PLHZTBOUQ266OIJW4WQMCPRQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:39.827220762 (60h51m49.549783802s) [upload=58359]: Next returned nil segment Jul 27 00:27:39.827239178 (60h51m49.549802218s) [upload=58359]: Waiting for error group managing segments... Jul 27 00:27:40.129200430 (60h51m49.85176347s) [noise=true , address=79.127.205.235:4668 , piece_id=6F5PMSOXC6DK2YLFXJFYPRI4NFTUYSGDISXUX5MBA4L7XHOTU6OA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:40.170677248 (60h51m49.893240288s) [noise=true , address=79.127.205.246:4275 , piece_id=JHTG62SBAUGKPOG2S5Q57H5IVZSZ4AL53AZSBJM2NLF6NEVQSOTA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:40.437354773 (60h51m50.159917813s) [noise=true , address=79.127.205.234:4616 , piece_id=FGIQTFKQHUEHCHBJ2N6BRGXFOI36QO46VKL3QIUDUVMMVRJ5ZTTQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:40.770782791 (60h51m50.493345831s) [noise=true , address=77.90.31.21:4905 , piece_id=CJOB6OCGNGQXPPXELLMKP22LMZJIJZAJ3U4FZBI7526CMAH7FABA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:41.162195952 (60h51m50.884758992s) [noise=true , address=79.127.219.38:4005 , piece_id=DR5TRODJ5CVJ4SQCEHICJP3UMDZVDMIHSNSRFEXJZAXZA6CNOP2A, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:42.093173773 (60h51m51.815736813s) [noise=true , address=77.90.31.12:4597 , piece_id=6ZQJO4W2HQKNYJV5WQRWZLRULTS4SDA45NDQ7PKHRDUC2LF7IBOQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:42.394361147 (60h51m52.116924187s) [noise=true , address=45.91.116.4:4964 , piece_id=HZ33J55E4NIGVAXCFEH6IYKCDVKWWAEBWPM5EQKK3L4TGEGT6G5Q, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:43.233840865 (60h51m52.956403905s) [noise=true , address=64.49.13.26:4953 , piece_id=H37PPSMDWBKRF4LUKWBAGZRQG4423L2GQTGXOPUU323RRACJHLQA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:43.253321728 (60h51m52.975884768s) [noise=true , address=64.49.13.138:4488 , piece_id=FPCV3SLHC5KPBO5LIUB4WTKE3HIK6SVPFLVSB7C5ZC4I22FDALUQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb, address:64.49.13.138:4488): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:11.280211356 (60h52m21.002774396s) [noise=true , address=77.90.31.13:4659 , piece_id=NZRBSSZVEKLDUKJB47TA77QEFZ5GVGJAGHA7CH3QCZ4KIK5IRHWQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:28:37.631488971 (60h52m47.354052001s) [noise=true , address=104.36.82.56:4235 , piece_id=Z4F3TG2WMMHBCDHNZOGZQP6UT25R4NQ6ZPZQWCLMKN4TFOXHQYEA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: write tcp 188.166.225.170:53922->104.36.82.56:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631531352 (60h52m47.354094392s) [noise=true , address=104.36.82.47:4518 , piece_id=OFRKVCFF7I42OTYMUGXMPQJKTMJWX2XBQ2TWEJIRIPP7H3VPJVBA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: rpc: dial tcp 104.36.82.47:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631562531 (60h52m47.354125572s) [noise=true , address=104.36.82.54:4181 , piece_id=MOMRZY5WTQN7BA7VNTL2MMYNSLDFGKVFHYBTYONSVCJLPQB4B7CA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12gGf5E7tzF2TXtLWgpK55t24SC1BpTSeC9G66nPmR1ARnDrAE8, address:104.36.82.54:4181): protocol: rpc: dial tcp 104.36.82.54:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631578252 (60h52m47.354141292s) [noise=true , address=104.36.82.55:4210 , piece_id=EVQD3W7LZ6ZE22WYOLYWEBKSDFFLTFDTVQ4CRFCWAMA5B45AXJ2A, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631612417 (60h52m47.354175457s) [noise=true , address=104.36.82.46:4498 , piece_id=5YJIJDWY62EP5B2NUKBF2UX7J2KVGHPZOYB6WT2W6FERLNXZL3UA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: write tcp 188.166.225.170:47506->104.36.82.46:4498: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631629029 (60h52m47.354192059s) [noise=true , address=77.90.31.16:4761 , piece_id=SSQPNG7UZS6GDFPWQLHTUIXBN44ZBBMN2QIGXEXYRVTFLGGMEAKA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631634750 (60h52m47.35419778s) [noise=true , address=104.36.82.52:4092 , piece_id=HPHY4OA25YF5DA55RTMOIH2V33BXQ34ORVOE5Z7C4J2KCEVNVPXA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: rpc: dial tcp 104.36.82.52:4092: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631642104 (60h52m47.354205144s) [noise=true , address=104.36.82.5:4866 , piece_id=RR6542LBU2TDSMU7UKHK45HYA4R2PLHZTBOUQ266OIJW4WQMCPRQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 188.166.225.170:51822->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631663766 (60h52m47.354226806s) [seg_pos={0 0}, upload=58359]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:28:37.868103125 (60h52m47.590666165s) [upload=58359]: Done uploading segments... Jul 27 00:28:37.868129075 (60h52m47.590692115s) [upload=58359]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.238555292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................................X.....] duration: 1.822820241s FAIL (2025-07-27 00:28:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.822847604s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X......X...XX..XX.X.XX...X...............XX......X...XXX..X] duration: 1m0.001143858s FAIL (2025-07-27 00:28:17 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001168375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXX] duration: 30.24284958s FAIL (2025-07-27 00:28:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 7 00:27:46.719859324 (60h51m56.442422364s) [upload=58361]: Done starting upload Jul 27 00:27:46.720002880 (60h51m56.44256592s) [upload=58361]: Uploading segments... Jul 27 00:27:46.720143799 (60h51m56.442706859s) [upload=58361]: Got next segment. Inline: false Jul 27 00:27:46.720191270 (60h51m56.4427543s) [upload=58361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:27:46.720196781 (60h51m56.442759821s) [upload=58361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:27:47.192266913 (60h51m56.914829953s) [seg_pos={0 0}, upload=58361]: Begin upload segment... Jul 27 00:27:47.192315045 (60h51m56.914878085s) [seg_pos={0 0}, upload=58361]: Done begin upload segment. Jul 27 00:27:47.192332157 (60h51m56.914895197s) [noise=true , address=45.91.116.12:4225 , piece_id=66HOLX2FMX4U4RSLGYKCROD425XUIXJHFQYUEUN47BUOGIE6DQBA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192339902 (60h51m56.914902942s) [noise=true , address=45.91.116.13:4153 , piece_id=5E7NKTJPDHGH7IWYSFE4CRNFWYLTJKRPGOCZEI653FR3DV3FQLVQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192703989 (60h51m56.915267029s) [noise=true , address=104.36.82.5:4837 , piece_id=N3KD6RHD7AXCXNF7TRN4IVLT6JK7FPEX4PNN6BQ33I6PLLOU2HFA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192712776 (60h51m56.915275806s) [noise=true , address=77.90.31.21:4878 , piece_id=GNROAAE6XLCVA7LITD6HNYA2BSBRH7VZSXMJCDPICV6L3C7VDTVA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192949249 (60h51m56.915512289s) [noise=true , address=104.36.82.11:4113 , piece_id=C4BTLU4X3PNKVPTFGN6WMSMZ5BSIK5LEGGO7765NSWASR5BRHKNA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.193188566 (60h51m56.915751606s) [noise=true , address=45.91.116.11:4118 , piece_id=45AMYZVDBEZSLUZZ7PEMTOXZDAH4DIBDXWPBNXDDIVAMBL46Q4RQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.193512175 (60h51m56.916075215s) [noise=true , address=104.36.82.15:4257 , piece_id=UMYK3FRUKCRPI55OCWKZTPTKOP2UZGVSSWCWYDSJ4N5SVPLLS4KA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.194339288 (60h51m56.916902318s) [noise=true , address=77.90.31.12:4590 , piece_id=MOQ2LPYGMUS24TIPDPWWVK5LFTOA7BVF5D5CR23WGXYPAURC3N3A, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.194627660 (60h51m56.9171907s) [noise=true , address=45.91.116.14:4190 , piece_id=WNMKGCRLIYUS2GU37HOWO4I3KTPEBIIUR64K6MZJSEHLNTEBH7ZQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.194868069 (60h51m56.917431109s) [noise=true , address=104.36.82.13:4187 , piece_id=M5YZD4PN5L3EKQE4EYPKPERJKLSVAGCYUOMVL64YWIQFE3AFEFIA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195086838 (60h51m56.917649878s) [noise=true , address=104.36.82.12:4150 , piece_id=A76BRUD5B3CXB7E22VEB4XNVS3EBKCYMU4V4YXZ63C3R2NSJGVOQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195306929 (60h51m56.917869969s) [noise=true , address=104.36.82.10:4077 , piece_id=DNXWQ2K4RPQBH7CBRGBNY3IYE34K5SOJUUR7563HC4VD4FG2LUQA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195548311 (60h51m56.918111351s) [noise=true , address=77.90.31.19:4842 , piece_id=KAWHNT2EGJB5LHJAGOHLAZGASMNFWSCRRRJVC3HGRQO7ILIQAZJA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195751811 (60h51m56.918314851s) [noise=true , address=77.90.31.10:4518 , piece_id=B66TNNWRDS5X62NBRVWOBPJ4NO3AULH5A5WMPDFDCGWIY3DJYSNA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.196038709 (60h51m56.918601749s) [noise=true , address=77.90.31.18:4805 , piece_id=S3RA6LIBKTSDOF2S4R52JMOFA6EQS5OBQ6ZAQMV27LPPFMC7OKHA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.196360034 (60h51m56.918923064s) [noise=true , address=104.36.82.9:4007 , piece_id=XFGF4XPGZYGI3QB2623ZIAIGNH7IJHWKM2DLXZ6B436HJEQ4DFVA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:48.933514472 (60h51m58.656077502s) [upload=58361]: Next returned nil segment Jul 27 00:27:48.933534531 (60h51m58.656097571s) [upload=58361]: Waiting for error group managing segments... Jul 27 00:27:49.218628530 (60h51m58.941191571s) [noise=true , address=45.91.116.13:4153 , piece_id=5E7NKTJPDHGH7IWYSFE4CRNFWYLTJKRPGOCZEI653FR3DV3FQLVQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.306065795 (60h51m59.028628835s) [noise=true , address=45.91.116.11:4118 , piece_id=45AMYZVDBEZSLUZZ7PEMTOXZDAH4DIBDXWPBNXDDIVAMBL46Q4RQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.373863643 (60h51m59.096426683s) [noise=true , address=45.91.116.12:4225 , piece_id=66HOLX2FMX4U4RSLGYKCROD425XUIXJHFQYUEUN47BUOGIE6DQBA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.401482766 (60h51m59.124045796s) [noise=true , address=45.91.116.14:4190 , piece_id=WNMKGCRLIYUS2GU37HOWO4I3KTPEBIIUR64K6MZJSEHLNTEBH7ZQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.403840978 (60h51m59.126404019s) [noise=true , address=77.90.31.21:4878 , piece_id=GNROAAE6XLCVA7LITD6HNYA2BSBRH7VZSXMJCDPICV6L3C7VDTVA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:50.618632772 (60h52m0.341195812s) [noise=true , address=77.90.31.12:4590 , piece_id=MOQ2LPYGMUS24TIPDPWWVK5LFTOA7BVF5D5CR23WGXYPAURC3N3A, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:50.737040051 (60h52m0.459603081s) [noise=true , address=77.90.31.18:4805 , piece_id=S3RA6LIBKTSDOF2S4R52JMOFA6EQS5OBQ6ZAQMV27LPPFMC7OKHA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:51.237771652 (60h52m0.960334692s) [noise=true , address=77.90.31.19:4842 , piece_id=KAWHNT2EGJB5LHJAGOHLAZGASMNFWSCRRRJVC3HGRQO7ILIQAZJA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:28:16.070387132 (60h52m25.792950172s) [noise=true , address=104.36.82.15:4257 , piece_id=UMYK3FRUKCRPI55OCWKZTPTKOP2UZGVSSWCWYDSJ4N5SVPLLS4KA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:45188->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070450563 (60h52m25.793013603s) [noise=true , address=104.36.82.11:4113 , piece_id=C4BTLU4X3PNKVPTFGN6WMSMZ5BSIK5LEGGO7765NSWASR5BRHKNA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:39334->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070517452 (60h52m25.793080482s) [noise=true , address=77.90.31.10:4518 , piece_id=B66TNNWRDS5X62NBRVWOBPJ4NO3AULH5A5WMPDFDCGWIY3DJYSNA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070579260 (60h52m25.7931423s) [noise=true , address=104.36.82.9:4007 , piece_id=XFGF4XPGZYGI3QB2623ZIAIGNH7IJHWKM2DLXZ6B436HJEQ4DFVA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:55948->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070606482 (60h52m25.793169522s) [noise=true , address=104.36.82.5:4837 , piece_id=N3KD6RHD7AXCXNF7TRN4IVLT6JK7FPEX4PNN6BQ33I6PLLOU2HFA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:52826->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070613044 (60h52m25.793176084s) [noise=true , address=104.36.82.13:4187 , piece_id=M5YZD4PN5L3EKQE4EYPKPERJKLSVAGCYUOMVL64YWIQFE3AFEFIA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070649525 (60h52m25.793212565s) [noise=true , address=104.36.82.12:4150 , piece_id=A76BRUD5B3CXB7E22VEB4XNVS3EBKCYMU4V4YXZ63C3R2NSJGVOQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070670444 (60h52m25.793233484s) [noise=true , address=104.36.82.10:4077 , piece_id=DNXWQ2K4RPQBH7CBRGBNY3IYE34K5SOJUUR7563HC4VD4FG2LUQA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070695342 (60h52m25.793258382s) [seg_pos={0 0}, upload=58361]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:28:16.311840156 (60h52m26.034403206s) [upload=58361]: Done uploading segments... Jul 27 00:28:16.311867668 (60h52m26.034430719s) [upload=58361]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.242877011s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X] duration: 30.001300254s FAIL (2025-07-27 00:27:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001323318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.] duration: 8.329090276s FAIL (2025-07-27 00:27:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 8.329105196s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.249543571s FAIL (2025-07-27 00:27:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=58345]: Got next segment. Inline: false Jul 27 00:26:37.803303071 (60h50m47.525866102s) [upload=58345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:26:37.803307209 (60h50m47.525870249s) [upload=58345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:26:38.247618197 (60h50m47.970181238s) [seg_pos={0 0}, upload=58345]: Begin upload segment... Jul 27 00:26:38.247668043 (60h50m47.970231083s) [seg_pos={0 0}, upload=58345]: Done begin upload segment. Jul 27 00:26:38.247677382 (60h50m47.970240422s) [noise=true , address=79.127.219.47:4048 , piece_id=C46EDMKQ3YBPBDDY62DFAYEXOH3W7PRIOJYYKWV4CRDRYK3ZTQFA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.247686148 (60h50m47.970249188s) [noise=true , address=156.146.43.225:4448 , piece_id=CYDRG2LGU6DTZP5XD7XMKQW2B2ZMJNMK3OF2MNMFGFXAKFVP6WFA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.247759318 (60h50m47.970322358s) [noise=true , address=45.91.116.9:4277 , piece_id=WXAU5XNITBIABAUJ754IAS44VJKRHTWY43BTWODCDOTBWPOHDDPA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248053851 (60h50m47.970616881s) [noise=true , address=77.90.31.19:4867 , piece_id=LBDWWQ44CQU6ADVQ5TLLZAGFECXS47CJ3J5U5TWT72MX55RGEUYQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248317205 (60h50m47.970880245s) [noise=true , address=79.127.205.234:4610 , piece_id=D5VNQZ7HJ2CGZEJE6EO45L4OYWI5QZBXYXGEON7VQRQ4C4OM5GEA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248573586 (60h50m47.971136626s) [noise=true , address=45.91.116.12:4231 , piece_id=TF7H6QABRIOBVQIMAOU7CDMOXTN4C53QLSP4RDBSKCVOXVQ5RBFQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248832231 (60h50m47.971395261s) [noise=true , address=45.91.116.14:4206 , piece_id=56HED72YCLGJ4Z4WTHKK2OK4Y3M2IETGJLBAITDEIVCLP7XDCFFQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249064495 (60h50m47.971627525s) [noise=true , address=77.90.31.11:4569 , piece_id=BJQ6OSLDCEVIYRWVXGSP5HXN5NX2GFT3DSIOJ3E7DARFRZCOCBHQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249256532 (60h50m47.971819573s) [noise=true , address=79.127.219.38:4006 , piece_id=GALBWCTGXIRYXUBUI6GQRX7FRA27UYRLBY26SV25RDUL3MRTUBDQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249457376 (60h50m47.972020417s) [noise=true , address=104.36.82.55:4218 , piece_id=EZD3JBEPP3WOB4DYN7LP5F6FZ2TDRLUQPMOIKFI3ZD52KUBSXVHQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249648733 (60h50m47.972211773s) [noise=true , address=104.36.82.6:4892 , piece_id=QTMLBADCPSU2RI6PY6UXQOWH7ZOFY52YNKYVHEI4CE66L2BD5CFA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249885095 (60h50m47.972448135s) [noise=true , address=77.90.31.21:4912 , piece_id=IEKAI3OWAJJRJZ5VEP2NFK5XIMRNT2LY7PMOQLFTVDZEFK3NYFZA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250074488 (60h50m47.972637528s) [noise=true , address=77.90.31.15:4715 , piece_id=X664TG2GDMUCYM7JEJIEJ2DN3JET5UQRELC4QHL3R3R3EYEVQUTA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250255283 (60h50m47.972818313s) [noise=true , address=79.127.205.245:4241 , piece_id=VNYMYPJKAPRUUSGJ2GCOU6TDUNHZX3EZUQ2KGQMTEOCE2RXAXNXA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250433996 (60h50m47.972997026s) [noise=true , address=64.49.13.26:4953 , piece_id=LHXBN46BWHQHLUPO5AQ2O3QWKQ4V67A55LXIOM7SIIR2UMGGOZOA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250623999 (60h50m47.973187039s) [noise=true , address=79.127.205.235:4679 , piece_id=ZPKL6OXQHJMW3KV6I2IT3N55ZO2EWV37KVA6IETE4AMSQQ2ZFWJA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250864849 (60h50m47.973427889s) [noise=true , address=104.36.82.47:4551 , piece_id=LOG3MYYZZCTPVB2TPAJIPUL5DMSPLW2WZLCK7O2NMGU4XX7TYMAA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.251066686 (60h50m47.973629726s) [noise=true , address=104.36.82.46:4493 , piece_id=ZEP2LHC3SUJJSYTVMA52WQNWRJY6RVCCHGRKQZG4MNT6OD4KGOKA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:39.348414773 (60h50m49.070977813s) [upload=58345]: Next returned nil segment Jul 27 00:26:39.348433138 (60h50m49.070996169s) [upload=58345]: Waiting for error group managing segments... Jul 27 00:26:39.646504488 (60h50m49.369067518s) [noise=true , address=79.127.205.245:4241 , piece_id=VNYMYPJKAPRUUSGJ2GCOU6TDUNHZX3EZUQ2KGQMTEOCE2RXAXNXA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:39.649598598 (60h50m49.372161638s) [noise=true , address=79.127.205.234:4610 , piece_id=D5VNQZ7HJ2CGZEJE6EO45L4OYWI5QZBXYXGEON7VQRQ4C4OM5GEA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:39.854697454 (60h50m49.577260494s) [noise=true , address=79.127.205.235:4679 , piece_id=ZPKL6OXQHJMW3KV6I2IT3N55ZO2EWV37KVA6IETE4AMSQQ2ZFWJA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.167500830 (60h50m49.89006386s) [noise=true , address=77.90.31.15:4715 , piece_id=X664TG2GDMUCYM7JEJIEJ2DN3JET5UQRELC4QHL3R3R3EYEVQUTA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.208961507 (60h50m49.931524537s) [noise=true , address=77.90.31.21:4912 , piece_id=IEKAI3OWAJJRJZ5VEP2NFK5XIMRNT2LY7PMOQLFTVDZEFK3NYFZA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.321108979 (60h50m50.043672019s) [noise=true , address=79.127.219.38:4006 , piece_id=GALBWCTGXIRYXUBUI6GQRX7FRA27UYRLBY26SV25RDUL3MRTUBDQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.349425747 (60h50m50.071988777s) [noise=true , address=64.49.13.26:4953 , piece_id=LHXBN46BWHQHLUPO5AQ2O3QWKQ4V67A55LXIOM7SIIR2UMGGOZOA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:26:42.739609688 (60h50m52.462172728s) [noise=true , address=156.146.43.225:4448 , piece_id=CYDRG2LGU6DTZP5XD7XMKQW2B2ZMJNMK3OF2MNMFGFXAKFVP6WFA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:43.703533724 (60h50m53.426096764s) [noise=true , address=45.91.116.9:4277 , piece_id=WXAU5XNITBIABAUJ754IAS44VJKRHTWY43BTWODCDOTBWPOHDDPA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:44.317755234 (60h50m54.040318264s) [noise=true , address=79.127.219.47:4048 , piece_id=C46EDMKQ3YBPBDDY62DFAYEXOH3W7PRIOJYYKWV4CRDRYK3ZTQFA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:44.731430629 (60h50m54.453993679s) [noise=true , address=45.91.116.12:4231 , piece_id=TF7H6QABRIOBVQIMAOU7CDMOXTN4C53QLSP4RDBSKCVOXVQ5RBFQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:46.165149866 (60h50m55.887712906s) [noise=true , address=45.91.116.14:4206 , piece_id=56HED72YCLGJ4Z4WTHKK2OK4Y3M2IETGJLBAITDEIVCLP7XDCFFQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:52.233095689 (60h51m1.955658729s) [noise=true , address=77.90.31.19:4867 , piece_id=LBDWWQ44CQU6ADVQ5TLLZAGFECXS47CJ3J5U5TWT72MX55RGEUYQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:27:37.162236246 (60h51m46.884799286s) [noise=true , address=104.36.82.46:4493 , piece_id=ZEP2LHC3SUJJSYTVMA52WQNWRJY6RVCCHGRKQZG4MNT6OD4KGOKA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162340596 (60h51m46.884903636s) [noise=true , address=77.90.31.11:4569 , piece_id=BJQ6OSLDCEVIYRWVXGSP5HXN5NX2GFT3DSIOJ3E7DARFRZCOCBHQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ, address:77.90.31.11:4569): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:27:37.162576166 (60h51m46.885139206s) [noise=true , address=104.36.82.6:4892 , piece_id=QTMLBADCPSU2RI6PY6UXQOWH7ZOFY52YNKYVHEI4CE66L2BD5CFA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:12MmZivyjXMMHcDgU3aG4HkWqBgY2QRf4Q1pkxJDF5QKSTQ7RXA, address:104.36.82.6:4892): protocol: write tcp 188.166.225.170:55824->104.36.82.6:4892: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162678031 (60h51m46.885241071s) [noise=true , address=104.36.82.47:4551 , piece_id=LOG3MYYZZCTPVB2TPAJIPUL5DMSPLW2WZLCK7O2NMGU4XX7TYMAA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:1UhJvSMtNqJDak4TM7eUgz1N7yCn4ChmRRTe7nXdffTNsS1pch, address:104.36.82.47:4551): protocol: rpc: dial tcp 104.36.82.47:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162800296 (60h51m46.885363336s) [noise=true , address=104.36.82.55:4218 , piece_id=EZD3JBEPP3WOB4DYN7LP5F6FZ2TDRLUQPMOIKFI3ZD52KUBSXVHQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:37466->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162819672 (60h51m46.885382702s) [seg_pos={0 0}, upload=58345]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:27:37.409519612 (60h51m47.132082652s) [upload=58345]: Done uploading segments... Jul 27 00:27:37.409543798 (60h51m47.132106839s) [upload=58345]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.249554743s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)