Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.847118933s FAIL (2025-07-27 00:09:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.847148731s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.............] duration: 52.860733119s FAIL (2025-07-27 00:09:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.860746493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX] duration: 30.264421826s FAIL (2025-07-27 00:09:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.264443898s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234980967s FAIL (2025-07-27 00:09:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=HUUZUMK2MXON23DYDTI5SAVL2PDGQCZJBFOSBBSVY3I5IXCRBXAA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191468653 (60h32m33.914031693s) [noise=true , address=79.127.219.38:4003 , piece_id=HGV4JIAWMIF52L6BMBE2JTBDSZXRKDPBRCIBFS72SDIKUL5X4QCQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191592921 (60h32m33.914155961s) [noise=true , address=45.91.116.9:4277 , piece_id=VI4FPX34RBSQBZVYJG6JEHWH3VDIZFD36TSEYKUWBC5MZTSCFD4A, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191758949 (60h32m33.914322019s) [noise=true , address=77.90.31.12:4608 , piece_id=PNSNCD462QGF44F57B5ABKWNAFXNXRRCIJOUPLQWWYR5UIYC3LQA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191896712 (60h32m33.914459752s) [noise=true , address=104.36.82.50:4016 , piece_id=WDZYETH3W6LW2T6UWYTQMN6SFLGHIAJWEDRAD2P3AE4A2UMNCPOA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192077898 (60h32m33.914640939s) [noise=true , address=79.127.220.101:4041 , piece_id=EMIQWCARLYTXACBD6O6DZH77RLYWX7EKA5PMFVNCJVQLCFW7WLFA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192227135 (60h32m33.914790175s) [noise=true , address=104.36.82.47:4532 , piece_id=BV2SJ3WCFJVTTO3A5QMITXSZVAMCQTKHJZOQQEQWVE5B4FUCHCJQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192366661 (60h32m33.914929702s) [noise=true , address=79.127.205.245:4250 , piece_id=KVMWKUPRKG2AD5YD2DLDBASPNKQWX3DVK4HPGJRTQE4HZEF36SZA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192522870 (60h32m33.91508591s) [noise=true , address=79.127.219.47:4048 , piece_id=LWIVS27EL4PHHBJ7W44KW3LNXSLK3VLXLGACZPIQZMIQVRLZ5HCA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192645184 (60h32m33.915208244s) [noise=true , address=79.127.205.235:4680 , piece_id=5SEJDTYH3PQAZSRZ3BNCILT4SPTDBN6F2I5CBM7XCF5HDFPYUU2A, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192798868 (60h32m33.915361908s) [noise=true , address=104.36.82.54:4192 , piece_id=ERPUZFYA2ND4TU5NBQPX22UW426C4VSOW2Z7TJBNYIMTAJUVVP4Q, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192909720 (60h32m33.91547276s) [noise=true , address=104.36.82.41:4783 , piece_id=FSSYD2VORTCAASQAK62RV4RFU5XFRJXNT4LTSRFQDCC5HJCNOUKQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193028888 (60h32m33.915591928s) [noise=true , address=104.36.82.46:4491 , piece_id=6ACSWEXX2EOI6Y2JHZIDTJ22AKUTUSZZUR7MCY4WE5XFEOWKHMQA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193189866 (60h32m33.915752906s) [noise=true , address=79.127.201.215:4356 , piece_id=DQHKW2AN3YEYLHOGW6JPUVLP4JWLCK5NFZ5AKIQRDQ3ZQZFOZYFQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193269258 (60h32m33.915832298s) [noise=true , address=45.91.116.10:4093 , piece_id=3AHXMLVONPDEVJLF7M7FKS4TWOU5HD3GJJIBHB5K5YQKC7O4A6HQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193446668 (60h32m33.916009708s) [noise=true , address=79.127.219.48:4087 , piece_id=RDDEOGNCM5DDUBZS7HPVJPKBK6GWKOHX7NCCS5GXROE3MXDUY2SA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193675024 (60h32m33.916238064s) [noise=true , address=156.146.43.225:4434 , piece_id=M4DFDGN23AS7FY326EBM57YNL3Y552CWGHOEDVUYLB5L25IECD7Q, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:25.135671862 (60h32m34.858234902s) [noise=true , address=79.127.220.101:4041 , piece_id=EMIQWCARLYTXACBD6O6DZH77RLYWX7EKA5PMFVNCJVQLCFW7WLFA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:08:25.229683572 (60h32m34.952246612s) [noise=true , address=79.127.219.47:4048 , piece_id=LWIVS27EL4PHHBJ7W44KW3LNXSLK3VLXLGACZPIQZMIQVRLZ5HCA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:08:25.308928970 (60h32m35.03149201s) [upload=58070]: Next returned nil segment Jul 27 00:08:25.308950642 (60h32m35.031513682s) [upload=58070]: Waiting for error group managing segments... Jul 27 00:08:25.596587471 (60h32m35.319150501s) [noise=true , address=79.127.205.235:4680 , piece_id=5SEJDTYH3PQAZSRZ3BNCILT4SPTDBN6F2I5CBM7XCF5HDFPYUU2A, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:25.596694865 (60h32m35.319257905s) [noise=true , address=79.127.205.245:4250 , piece_id=KVMWKUPRKG2AD5YD2DLDBASPNKQWX3DVK4HPGJRTQE4HZEF36SZA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:26.179102912 (60h32m35.901665952s) [noise=true , address=79.127.219.38:4003 , piece_id=HGV4JIAWMIF52L6BMBE2JTBDSZXRKDPBRCIBFS72SDIKUL5X4QCQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:26.303026391 (60h32m36.025589431s) [noise=true , address=79.127.219.48:4087 , piece_id=RDDEOGNCM5DDUBZS7HPVJPKBK6GWKOHX7NCCS5GXROE3MXDUY2SA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:29.930039220 (60h32m39.65260226s) [noise=true , address=45.91.116.10:4093 , piece_id=3AHXMLVONPDEVJLF7M7FKS4TWOU5HD3GJJIBHB5K5YQKC7O4A6HQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:29.939541440 (60h32m39.66210449s) [noise=true , address=77.90.31.12:4608 , piece_id=PNSNCD462QGF44F57B5ABKWNAFXNXRRCIJOUPLQWWYR5UIYC3LQA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:30.410036577 (60h32m40.132599657s) [noise=true , address=156.146.43.225:4434 , piece_id=M4DFDGN23AS7FY326EBM57YNL3Y552CWGHOEDVUYLB5L25IECD7Q, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:35.944972414 (60h32m45.667535484s) [noise=true , address=45.91.116.9:4277 , piece_id=VI4FPX34RBSQBZVYJG6JEHWH3VDIZFD36TSEYKUWBC5MZTSCFD4A, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:59.456991076 (60h33m9.179554116s) [noise=true , address=79.127.201.215:4356 , piece_id=DQHKW2AN3YEYLHOGW6JPUVLP4JWLCK5NFZ5AKIQRDQ3ZQZFOZYFQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:09:23.094493363 (60h33m32.817056393s) [noise=true , address=104.36.82.54:4192 , piece_id=ERPUZFYA2ND4TU5NBQPX22UW426C4VSOW2Z7TJBNYIMTAJUVVP4Q, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094541304 (60h33m32.817104344s) [noise=true , address=104.36.82.50:4016 , piece_id=WDZYETH3W6LW2T6UWYTQMN6SFLGHIAJWEDRAD2P3AE4A2UMNCPOA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR, address:104.36.82.50:4016): protocol: write tcp 188.166.225.170:59734->104.36.82.50:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094859924 (60h33m32.817422994s) [noise=true , address=104.36.82.47:4532 , piece_id=BV2SJ3WCFJVTTO3A5QMITXSZVAMCQTKHJZOQQEQWVE5B4FUCHCJQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 188.166.225.170:41088->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094920129 (60h33m32.817483219s) [noise=true , address=104.36.82.46:4491 , piece_id=6ACSWEXX2EOI6Y2JHZIDTJ22AKUTUSZZUR7MCY4WE5XFEOWKHMQA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: write tcp 188.166.225.170:51990->104.36.82.46:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095027074 (60h33m32.817590154s) [noise=true , address=104.36.82.55:4213 , piece_id=XAY5SRUIMYDFPKPWZTBSVHCNFSAVMOPKOIB7ZZ6AJEPN7WNB4AAA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095099132 (60h33m32.817662172s) [noise=true , address=104.36.82.41:4783 , piece_id=FSSYD2VORTCAASQAK62RV4RFU5XFRJXNT4LTSRFQDCC5HJCNOUKQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: write tcp 188.166.225.170:54126->104.36.82.41:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095202749 (60h33m32.817765819s) [noise=true , address=104.36.82.10:4094 , piece_id=HUUZUMK2MXON23DYDTI5SAVL2PDGQCZJBFOSBBSVY3I5IXCRBXAA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: write tcp 188.166.225.170:48722->104.36.82.10:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095250731 (60h33m32.817813791s) [seg_pos={0 0}, upload=58070]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:09:23.327131253 (60h33m33.049694293s) [upload=58070]: Done uploading segments... Jul 27 00:09:23.327157522 (60h33m33.049720562s) [upload=58070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.234997437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...X] duration: 30.000883423s FAIL (2025-07-27 00:09:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000905465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..] duration: 7.289167177s FAIL (2025-07-27 00:09:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.289196683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..XXXX.....XXX..X.....X......X...XX..XX.X.XX...X............] duration: 47.107254625s FAIL (2025-07-27 00:08:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.107266408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.872802594s FAIL (2025-07-27 00:08:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.872831339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXX] duration: 30.239811509s FAIL (2025-07-27 00:08:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.239842867s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.817189051s FAIL (2025-07-27 00:08:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.81722028s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244427209s FAIL (2025-07-27 00:08:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ddress=64.49.13.139:4478 , piece_id=6UJX4IIE6CYNLDMRUJ6BHLD65WT3UXZFVNRPD6MEVIUHPR6GXSAA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727061052 (60h31m33.449624122s) [noise=true , address=79.127.220.97:4361 , piece_id=TYJP6IPDMMHMERFCFJNL4NXJZZCG2UYNS2KB3JPGXWXEPINXJ2PA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727196460 (60h31m33.44975955s) [noise=true , address=77.90.31.15:4726 , piece_id=7KKKQMUN7BQHWDMCJTPGNKDKPFGUR33HY4CBCHH343ICI467ROXA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727682550 (60h31m33.45024561s) [noise=true , address=104.36.82.54:4191 , piece_id=A7MNSY6RGTLUT6V46SLEEAQOCPXDRY5QFXBCQOAZ2KHUBQY73SWA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728099728 (60h31m33.450662758s) [noise=true , address=104.36.82.46:4493 , piece_id=KTQPHH3XGIMXINOQRHSVA7ZY2VBOHTFI6LLHGUDBO5X5JBKVJE6A, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728106522 (60h31m33.450669562s) [noise=true , address=79.127.205.235:4687 , piece_id=3J6XO5R7RAKTZZVUSTCQ5S545NDCFCLXO3PBFSHXFWSZANLNDLAA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728548447 (60h31m33.451111487s) [noise=true , address=104.36.82.55:4204 , piece_id=VYOPSSZNCYIO2NE5GOSKVWOEDXSNCMXDLIEHMHSJYXF6KQHAE6QA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.729766277 (60h31m33.452329317s) [noise=true , address=79.127.219.38:4003 , piece_id=N5R7ILGBGWODMYVA72UKNSBLHO2ZCZOHPVGIKSSRILX2TDNAK6QA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.730872413 (60h31m33.453435453s) [noise=true , address=104.36.82.16:4300 , piece_id=SW3L2SBYEEUMIEPT6BNUUBGYLBGZW4CB3KZUWVER2MNHZ3GH6HEA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.731313136 (60h31m33.453876177s) [noise=true , address=77.90.31.16:4737 , piece_id=H3GQJUAD3HGZLR4UFSVWNQDWWYFQGABYWJYWQPZULNFETEN2V6MA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.731703463 (60h31m33.454266503s) [noise=true , address=104.36.82.53:4142 , piece_id=WMZCU4FYC4KXVD5TO7PQ5S45NSSJL4IN45UNRBZTUZ5N6MBA35WA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732105703 (60h31m33.454668743s) [noise=true , address=79.127.205.234:4610 , piece_id=GBFRWZZDAX2SUH55K7SHR4MBTV5ADHFH26JOR2F7D5R5VWVFKVTA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732370520 (60h31m33.45493357s) [noise=true , address=79.127.219.47:4048 , piece_id=DS26KSX25RBT3DEEKXDFNXCROHU2AINQKMMS5TN6DJMTNVVUA56Q, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732647630 (60h31m33.45521067s) [noise=true , address=79.127.205.246:4270 , piece_id=AY4DVMJGAVSKY5C2ZP4WAAQZPRA2KNHP6HBSTJHY6OFL3MT5NCNA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732921374 (60h31m33.455484414s) [noise=true , address=45.91.116.10:4093 , piece_id=USC6TAFNA35EBKBQLDEJRSIYUESRZWSEI6SMICMWISZY7U7R7RHA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.733277195 (60h31m33.455840225s) [noise=true , address=79.127.201.215:4355 , piece_id=RV2YPT7IJ5TRCXJMXCWBZTHS43ME42WUFRYUF3LLD6FNKI2BEYIA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:24.820008849 (60h31m34.542571889s) [upload=58052]: Next returned nil segment Jul 27 00:07:24.820025310 (60h31m34.542588351s) [upload=58052]: Waiting for error group managing segments... Jul 27 00:07:24.887653151 (60h31m34.610216191s) [noise=true , address=79.127.219.47:4048 , piece_id=DS26KSX25RBT3DEEKXDFNXCROHU2AINQKMMS5TN6DJMTNVVUA56Q, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:25.109410853 (60h31m34.831973893s) [noise=true , address=79.127.205.234:4610 , piece_id=GBFRWZZDAX2SUH55K7SHR4MBTV5ADHFH26JOR2F7D5R5VWVFKVTA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.138673642 (60h31m34.861236682s) [noise=true , address=79.127.205.235:4687 , piece_id=3J6XO5R7RAKTZZVUSTCQ5S545NDCFCLXO3PBFSHXFWSZANLNDLAA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.153927788 (60h31m34.876490828s) [noise=true , address=79.127.205.246:4270 , piece_id=AY4DVMJGAVSKY5C2ZP4WAAQZPRA2KNHP6HBSTJHY6OFL3MT5NCNA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.642669774 (60h31m35.365232814s) [noise=true , address=79.127.220.97:4361 , piece_id=TYJP6IPDMMHMERFCFJNL4NXJZZCG2UYNS2KB3JPGXWXEPINXJ2PA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.719944903 (60h31m35.442507943s) [noise=true , address=79.127.219.38:4003 , piece_id=N5R7ILGBGWODMYVA72UKNSBLHO2ZCZOHPVGIKSSRILX2TDNAK6QA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.757257171 (60h31m35.479820201s) [noise=true , address=77.90.31.16:4737 , piece_id=H3GQJUAD3HGZLR4UFSVWNQDWWYFQGABYWJYWQPZULNFETEN2V6MA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:35.745696491 (60h31m45.468259532s) [noise=true , address=64.49.13.139:4478 , piece_id=6UJX4IIE6CYNLDMRUJ6BHLD65WT3UXZFVNRPD6MEVIUHPR6GXSAA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:40.589949786 (60h31m50.312512837s) [noise=true , address=45.91.116.10:4093 , piece_id=USC6TAFNA35EBKBQLDEJRSIYUESRZWSEI6SMICMWISZY7U7R7RHA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:47.462943792 (60h31m57.185506833s) [noise=true , address=79.127.201.215:4355 , piece_id=RV2YPT7IJ5TRCXJMXCWBZTHS43ME42WUFRYUF3LLD6FNKI2BEYIA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:08:22.624850013 (60h32m32.347413073s) [noise=true , address=77.90.31.15:4726 , piece_id=7KKKQMUN7BQHWDMCJTPGNKDKPFGUR33HY4CBCHH343ICI467ROXA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: write tcp 188.166.225.170:49330->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624893377 (60h32m32.347456417s) [noise=true , address=104.36.82.55:4204 , piece_id=VYOPSSZNCYIO2NE5GOSKVWOEDXSNCMXDLIEHMHSJYXF6KQHAE6QA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:44342->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624933333 (60h32m32.347496373s) [noise=true , address=104.36.82.16:4300 , piece_id=SW3L2SBYEEUMIEPT6BNUUBGYLBGZW4CB3KZUWVER2MNHZ3GH6HEA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg, address:104.36.82.16:4300): protocol: write tcp 188.166.225.170:38544->104.36.82.16:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624963520 (60h32m32.34752656s) [noise=true , address=104.36.82.54:4191 , piece_id=A7MNSY6RGTLUT6V46SLEEAQOCPXDRY5QFXBCQOAZ2KHUBQY73SWA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: write tcp 188.166.225.170:53522->104.36.82.54:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624996474 (60h32m32.347559554s) [noise=true , address=104.36.82.46:4493 , piece_id=KTQPHH3XGIMXINOQRHSVA7ZY2VBOHTFI6LLHGUDBO5X5JBKVJE6A, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 188.166.225.170:59700->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625029076 (60h32m32.347592277s) [noise=true , address=104.36.82.47:4526 , piece_id=6DBPGXO5PJCFQ7PKXOZK2IM6AELQQ7IC2MKVDWPPASOYSKX7PMPQ, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:48354->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625062891 (60h32m32.347626101s) [noise=true , address=104.36.82.53:4142 , piece_id=WMZCU4FYC4KXVD5TO7PQ5S45NSSJL4IN45UNRBZTUZ5N6MBA35WA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: write tcp 188.166.225.170:54566->104.36.82.53:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625098859 (60h32m32.347661899s) [noise=true , address=104.36.82.2:4736 , piece_id=ICMVAPJZ3HNI6AGWXV6NYKA5Y7W7IFOT2IFVGVWSNEQTNGWTDIGQ, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:104.36.82.2:4736): protocol: write tcp 188.166.225.170:57946->104.36.82.2:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625117215 (60h32m32.347680255s) [seg_pos={0 0}, upload=58052]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:08:22.867247115 (60h32m32.589810155s) [upload=58052]: Done uploading segments... Jul 27 00:08:22.867270961 (60h32m32.589834001s) [upload=58052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.24443828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XXXX.....XXX..X.....X......X...XX..XX.X.XX...X...........] duration: 50.659346226s FAIL (2025-07-27 00:08:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.659380051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...] duration: 7.359180024s FAIL (2025-07-27 00:08:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.359199812s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.631167904s FAIL (2025-07-27 00:07:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.631192571s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX.] duration: 7.190647044s FAIL (2025-07-27 00:07:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.190664127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX] duration: 30.235801722s FAIL (2025-07-27 00:07:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235823725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXX] duration: 30.241106914s FAIL (2025-07-27 00:07:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24111975s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX..] duration: 5.897942536s FAIL (2025-07-27 00:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.897964458s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.862636221s FAIL (2025-07-27 00:07:23 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.862666029s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231435993s FAIL (2025-07-27 00:07:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== item: pb.BatchRequestItem_SegmentBegin Jul 27 00:06:23.264389462 (60h30m32.986952502s) [seg_pos={0 0}, upload=58035]: Begin upload segment... Jul 27 00:06:23.264439347 (60h30m32.987002387s) [seg_pos={0 0}, upload=58035]: Done begin upload segment. Jul 27 00:06:23.264466770 (60h30m32.98702982s) [noise=true , address=79.127.219.38:4009 , piece_id=UFE4OFSPUN7UGZPMHYGCXVW5FIN7DGYCJYTE7CSP2QVQCI5HI2YQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.264525191 (60h30m32.987088231s) [noise=true , address=104.36.82.47:4552 , piece_id=QXIPZPL2A7KHS6IRFJI7HC47DYLRX2QTYCO5T3PZJCFA2LP44WPQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.264869410 (60h30m32.987432451s) [noise=true , address=104.36.82.46:4497 , piece_id=3YHTH7QRCV3AZZXJQF3NPABNSLZ3FNK3EC73MXAGUCYCZGTXGRGA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.264989820 (60h30m32.987552861s) [noise=true , address=79.127.219.51:4212 , piece_id=RU7AVMQFKGLT4UV7USAWN4UABHH74EDX52URHTEMHGJDAE73H37A, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265183531 (60h30m32.987746571s) [noise=true , address=77.90.31.12:4611 , piece_id=SLA2SPTQIK3ZIVKUZVHMWF26NFIOYOWYOARZRT7TNRVKRUKNYM7Q, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265344980 (60h30m32.98790802s) [noise=true , address=64.49.13.26:4953 , piece_id=ZQM5HCJHVI3JRM76N3KVIDIWVKDXSVUZAXX5LOHX6XW7UD5R4XTQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265523262 (60h30m32.988086302s) [noise=true , address=104.36.82.53:4124 , piece_id=F5HKMKDRGAAUYTVE3T45MXPKKPHZG3J6RI4NQZKARBILA3HSBRAQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265619225 (60h30m32.988182265s) [noise=true , address=104.36.82.54:4169 , piece_id=62OVJCI54IJRCDUJW7H5EMEKM6AOCXQELRSZZ3APAJIMLWGOFOPA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.265851629 (60h30m32.988414669s) [noise=true , address=77.90.31.11:4579 , piece_id=ZKEVLZTA62BYPOUR4RZOK236EAH7S5KK3HVRRFTPD7KOGGGKRFBQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266138869 (60h30m32.988701909s) [noise=true , address=79.127.201.215:4359 , piece_id=G72XJNM23VWHUURJ56JSFDR7G3JTBHPRAMUEIFITB62GPBHDYTTQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266410870 (60h30m32.98897391s) [noise=true , address=104.36.82.56:4264 , piece_id=FFWW5EL7S2A7H7D3VCZYBZTIRHGYGKIY2SQQLYVA6NN3XEDG35OQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266707317 (60h30m32.989270357s) [noise=true , address=79.127.205.246:4288 , piece_id=UGXYSTWYQL6INNZXXFN43QV2MQ4F4RYBP7BJ5UNQU3RUY36EDICA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266724219 (60h30m32.989287259s) [noise=true , address=77.90.31.13:4644 , piece_id=KJ4ARAULFCNVJJUH7PDAU54VZWDTQFDEQM2CULAJGBQMKF2MVUAA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266955372 (60h30m32.989518412s) [noise=true , address=79.127.205.234:4614 , piece_id=Y2DYAZYKYVWRWMQMPBHC3VEDQWI6DPXUN4KFCV47VJXX3WCLC7UA, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.266990118 (60h30m32.989553158s) [noise=true , address=64.49.13.138:4483 , piece_id=M2S7IXL2YCAVIXE3QM3S6YQAC3WCJQSURGFU6EJEXUMERE5JNP4A, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.267183307 (60h30m32.989746348s) [noise=true , address=156.146.43.225:4448 , piece_id=WVEOI6VV55WP5FH7ZMNCRG5P6DSYXQP5UTUOR4M5LEZ37V3MP6AQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.267276024 (60h30m32.989839065s) [noise=true , address=79.127.220.101:4074 , piece_id=FHWYUCW5CPJXAP4OQOQTSHFVA4NIQU7OYMYWHLP5ONBRFBEVI4GQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:23.267400734 (60h30m32.989963784s) [noise=true , address=77.90.31.15:4713 , piece_id=VVL6ZJFTPAEOSPLTKSGPZXNLEO5N2446KPBROMZ6ODPKFKF46SPQ, seg_pos={0 0}, upload=58035]: Uploading piece... Jul 27 00:06:24.387683351 (60h30m34.110246391s) [upload=58035]: Next returned nil segment Jul 27 00:06:24.387697508 (60h30m34.110260548s) [upload=58035]: Waiting for error group managing segments... Jul 27 00:06:24.673095542 (60h30m34.395658582s) [noise=true , address=79.127.205.246:4288 , piece_id=UGXYSTWYQL6INNZXXFN43QV2MQ4F4RYBP7BJ5UNQU3RUY36EDICA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:24.707938928 (60h30m34.430502028s) [noise=true , address=79.127.205.234:4614 , piece_id=Y2DYAZYKYVWRWMQMPBHC3VEDQWI6DPXUN4KFCV47VJXX3WCLC7UA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:24.740682445 (60h30m34.463245485s) [noise=true , address=64.49.13.26:4953 , piece_id=ZQM5HCJHVI3JRM76N3KVIDIWVKDXSVUZAXX5LOHX6XW7UD5R4XTQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:25.403201396 (60h30m35.125764436s) [noise=true , address=77.90.31.11:4579 , piece_id=ZKEVLZTA62BYPOUR4RZOK236EAH7S5KK3HVRRFTPD7KOGGGKRFBQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:25.413026002 (60h30m35.135589042s) [noise=true , address=79.127.219.38:4009 , piece_id=UFE4OFSPUN7UGZPMHYGCXVW5FIN7DGYCJYTE7CSP2QVQCI5HI2YQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:27.646840978 (60h30m37.369404018s) [noise=true , address=77.90.31.13:4644 , piece_id=KJ4ARAULFCNVJJUH7PDAU54VZWDTQFDEQM2CULAJGBQMKF2MVUAA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:27.779892114 (60h30m37.502455154s) [noise=true , address=156.146.43.225:4448 , piece_id=WVEOI6VV55WP5FH7ZMNCRG5P6DSYXQP5UTUOR4M5LEZ37V3MP6AQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:28.688597737 (60h30m38.411160777s) [noise=true , address=79.127.219.51:4212 , piece_id=RU7AVMQFKGLT4UV7USAWN4UABHH74EDX52URHTEMHGJDAE73H37A, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:29.067767433 (60h30m38.790330473s) [noise=true , address=79.127.220.101:4074 , piece_id=FHWYUCW5CPJXAP4OQOQTSHFVA4NIQU7OYMYWHLP5ONBRFBEVI4GQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:29.721835031 (60h30m39.444398071s) [noise=true , address=77.90.31.12:4611 , piece_id=SLA2SPTQIK3ZIVKUZVHMWF26NFIOYOWYOARZRT7TNRVKRUKNYM7Q, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:29.930228928 (60h30m39.652791968s) [noise=true , address=64.49.13.138:4483 , piece_id=M2S7IXL2YCAVIXE3QM3S6YQAC3WCJQSURGFU6EJEXUMERE5JNP4A, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:06:33.298983067 (60h30m43.021546097s) [noise=true , address=79.127.201.215:4359 , piece_id=G72XJNM23VWHUURJ56JSFDR7G3JTBHPRAMUEIFITB62GPBHDYTTQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: <nil> Jul 27 00:07:22.175214675 (60h31m31.897777715s) [noise=true , address=104.36.82.53:4124 , piece_id=F5HKMKDRGAAUYTVE3T45MXPKKPHZG3J6RI4NQZKARBILA3HSBRAQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175526061 (60h31m31.898089101s) [noise=true , address=104.36.82.54:4169 , piece_id=62OVJCI54IJRCDUJW7H5EMEKM6AOCXQELRSZZ3APAJIMLWGOFOPA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: write tcp 188.166.225.170:34160->104.36.82.54:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175557291 (60h31m31.898120332s) [noise=true , address=104.36.82.46:4497 , piece_id=3YHTH7QRCV3AZZXJQF3NPABNSLZ3FNK3EC73MXAGUCYCZGTXGRGA, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:12Zg4PSYBuLxFvF9oYiuvn65JNbjT5RdagcVAQLp2izvjPe1NEc, address:104.36.82.46:4497): protocol: write tcp 188.166.225.170:55572->104.36.82.46:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175591988 (60h31m31.898155028s) [noise=true , address=104.36.82.56:4264 , piece_id=FFWW5EL7S2A7H7D3VCZYBZTIRHGYGKIY2SQQLYVA6NN3XEDG35OQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 188.166.225.170:45042->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175607877 (60h31m31.898170918s) [noise=true , address=77.90.31.15:4713 , piece_id=VVL6ZJFTPAEOSPLTKSGPZXNLEO5N2446KPBROMZ6ODPKFKF46SPQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: write tcp 188.166.225.170:44700->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175623498 (60h31m31.898186538s) [noise=true , address=104.36.82.47:4552 , piece_id=QXIPZPL2A7KHS6IRFJI7HC47DYLRX2QTYCO5T3PZJCFA2LP44WPQ, seg_pos={0 0}, upload=58035]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: write tcp 188.166.225.170:60470->104.36.82.47:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:22.175647133 (60h31m31.898210163s) [seg_pos={0 0}, upload=58035]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:07:22.404227229 (60h31m32.126790269s) [upload=58035]: Done uploading segments... Jul 27 00:07:22.404245834 (60h31m32.126808864s) [upload=58035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.231446201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX.] duration: 7.576120302s FAIL (2025-07-27 00:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.576143827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXX] duration: 30.247039811s FAIL (2025-07-27 00:07:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.247060992s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX] duration: 30.000991283s FAIL (2025-07-27 00:07:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001000911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.836285196s FAIL (2025-07-27 00:06:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.836308362s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..........] duration: 30.79533043s FAIL (2025-07-27 00:06:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.795354036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.X] duration: 30.00191878s FAIL (2025-07-27 00:06:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00194941s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XX] duration: 30.239745298s FAIL (2025-07-27 00:06:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.239761157s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXX] duration: 30.231596387s FAIL (2025-07-27 00:06:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== s (needed at least 13 but got 10) Jul 27 00:05:57.436782393 (60h30m7.159345433s) [upload=58029]: Starting upload Jul 27 00:05:57.436841255 (60h30m7.159404295s) [upload=58029]: Done starting upload Jul 27 00:05:57.436944794 (60h30m7.159507834s) [upload=58029]: Uploading segments... Jul 27 00:05:57.437036960 (60h30m7.1596s) [upload=58029]: Got next segment. Inline: false Jul 27 00:05:57.437078049 (60h30m7.159641089s) [upload=58029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:05:57.437083178 (60h30m7.159646218s) [upload=58029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:05:57.878627803 (60h30m7.601190843s) [seg_pos={0 0}, upload=58029]: Begin upload segment... Jul 27 00:05:57.878668642 (60h30m7.601231682s) [seg_pos={0 0}, upload=58029]: Done begin upload segment. Jul 27 00:05:57.878693149 (60h30m7.601256189s) [noise=true , address=45.91.116.14:4191 , piece_id=XISWZZRL2XMU2IK5CBH7PVCPTTTSBB7VZP6NOCM4FSQAUDWES5DQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.878743705 (60h30m7.601306745s) [noise=true , address=45.91.116.7:4009 , piece_id=Q7MQKNWWMILXHKEOTXQG3IPSPPJPFMZPVIBLWNYSHNJFJUKOOLYA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879047837 (60h30m7.601610877s) [noise=true , address=77.90.31.17:4772 , piece_id=3RWZFAOC7H2IVAJ4ZELCJYDJ656LKP76FNFGIE4DUSVNZVUBJMVQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879220728 (60h30m7.601783768s) [noise=true , address=104.36.82.7:4910 , piece_id=C5LIRU4ITVNP5CAZRXWUQTDJ5EJDX2MPXPMI3YVEBQYJDFYVZE3A, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879296172 (60h30m7.601859212s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879559025 (60h30m7.602122065s) [noise=true , address=45.91.116.5:4299 , piece_id=TNZQNORDMODB4JVAOJNTWFIEDHYDHHMK3HXY3H7KKMM5IHO6YSXA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.879806168 (60h30m7.602369208s) [noise=true , address=104.36.82.7:4911 , piece_id=CQRJMWLXF2IZ3ZAZOS27SUO2KXD6ZGGXCPKAGIKFNB556TG37JRQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880097045 (60h30m7.602660085s) [noise=true , address=104.36.82.8:4971 , piece_id=KCRFRE2KRFTDFPUF2I5TXMP3AN7YVI543NPMGHGSWKGNL25DZY2A, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880331633 (60h30m7.602894673s) [noise=true , address=45.91.116.9:4262 , piece_id=PNUIEBHBU6HZEJ6EW6S6NRKATQJTML2GJ6IODKV5FPFQ76BEORJA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880595328 (60h30m7.603158368s) [noise=true , address=77.90.31.18:4807 , piece_id=HETSBBVKXOLAUTOUUIYFVGJIXR4UNDSYOHUKQIFVETKJV2KV4TTA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.880811973 (60h30m7.603375003s) [noise=true , address=104.36.82.12:4150 , piece_id=KTI4HOHOLP5CF3JCPUZYFPSMSASXJ72NPGRF65NZRN55LA4ECNZQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881019630 (60h30m7.60358267s) [noise=true , address=45.91.116.10:4081 , piece_id=H7M6NVHVBY6IBEXNZCE325DMP7SHHXAJ46AWBHE4D3B3ZOPIBIFA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881231736 (60h30m7.603794776s) [noise=true , address=77.90.31.15:4698 , piece_id=RUG7HTLJXQYCBHXMSK2UJH6BA3BXKCMGLAHUAPELUK4MLOLT3GFA, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881410217 (60h30m7.603973257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZNBUOT4CM6AUELDCJZULZRMYY3DPKAQGGAWZ4J47HYZSUL7R2YTQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881641800 (60h30m7.60420483s) [noise=true , address=104.36.82.6:4876 , piece_id=TIXMXTTEYPA57JH6J3G2CIED4ZLS2JLG5KMX5BLKU4HWX5WIHUKQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:57.881728897 (60h30m7.604291927s) [noise=true , address=45.91.116.13:4154 , piece_id=YFBPQS3DQAMJYHT5H32JWZUOVA5MIPJGUIGSVNPGXFKWLBYYGOXQ, seg_pos={0 0}, upload=58029]: Uploading piece... Jul 27 00:05:59.620668084 (60h30m9.343231124s) [upload=58029]: Next returned nil segment Jul 27 00:05:59.620685557 (60h30m9.343248597s) [upload=58029]: Waiting for error group managing segments... Jul 27 00:05:59.897369139 (60h30m9.619932179s) [noise=true , address=45.91.116.13:4154 , piece_id=YFBPQS3DQAMJYHT5H32JWZUOVA5MIPJGUIGSVNPGXFKWLBYYGOXQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.913229704 (60h30m9.635792744s) [noise=true , address=45.91.116.7:4009 , piece_id=Q7MQKNWWMILXHKEOTXQG3IPSPPJPFMZPVIBLWNYSHNJFJUKOOLYA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.976379096 (60h30m9.698942136s) [noise=true , address=45.91.116.10:4081 , piece_id=H7M6NVHVBY6IBEXNZCE325DMP7SHHXAJ46AWBHE4D3B3ZOPIBIFA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.983423965 (60h30m9.705987005s) [noise=true , address=77.90.31.14:4662 , piece_id=ZNBUOT4CM6AUELDCJZULZRMYY3DPKAQGGAWZ4J47HYZSUL7R2YTQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:05:59.998286812 (60h30m9.720849852s) [noise=true , address=45.91.116.5:4299 , piece_id=TNZQNORDMODB4JVAOJNTWFIEDHYDHHMK3HXY3H7KKMM5IHO6YSXA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:00.008836206 (60h30m9.731399236s) [noise=true , address=77.90.31.18:4807 , piece_id=HETSBBVKXOLAUTOUUIYFVGJIXR4UNDSYOHUKQIFVETKJV2KV4TTA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:00.124184478 (60h30m9.846747508s) [noise=true , address=45.91.116.9:4262 , piece_id=PNUIEBHBU6HZEJ6EW6S6NRKATQJTML2GJ6IODKV5FPFQ76BEORJA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:00.210814585 (60h30m9.933377625s) [noise=true , address=45.91.116.14:4191 , piece_id=XISWZZRL2XMU2IK5CBH7PVCPTTTSBB7VZP6NOCM4FSQAUDWES5DQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: <nil> Jul 27 00:06:26.796472853 (60h30m36.519035893s) [noise=true , address=77.90.31.17:4772 , piece_id=3RWZFAOC7H2IVAJ4ZELCJYDJ656LKP76FNFGIE4DUSVNZVUBJMVQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796538539 (60h30m36.519101579s) [noise=true , address=104.36.82.8:4971 , piece_id=KCRFRE2KRFTDFPUF2I5TXMP3AN7YVI543NPMGHGSWKGNL25DZY2A, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796576962 (60h30m36.519140002s) [noise=true , address=104.36.82.7:4911 , piece_id=CQRJMWLXF2IZ3ZAZOS27SUO2KXD6ZGGXCPKAGIKFNB556TG37JRQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:56848->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796622630 (60h30m36.51918567s) [noise=true , address=77.90.31.15:4698 , piece_id=RUG7HTLJXQYCBHXMSK2UJH6BA3BXKCMGLAHUAPELUK4MLOLT3GFA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:45116->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796661484 (60h30m36.519224524s) [noise=true , address=104.36.82.12:4150 , piece_id=KTI4HOHOLP5CF3JCPUZYFPSMSASXJ72NPGRF65NZRN55LA4ECNZQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796689097 (60h30m36.519252137s) [noise=true , address=104.36.82.6:4876 , piece_id=TIXMXTTEYPA57JH6J3G2CIED4ZLS2JLG5KMX5BLKU4HWX5WIHUKQ, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796710368 (60h30m36.519273408s) [noise=true , address=104.36.82.7:4910 , piece_id=C5LIRU4ITVNP5CAZRXWUQTDJ5EJDX2MPXPMI3YVEBQYJDFYVZE3A, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.231616634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.824645846s FAIL (2025-07-27 00:06:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.82466946s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.246366359s FAIL (2025-07-27 00:06:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== (60h29m32.51017865s) [noise=true , address=45.91.116.14:4206 , piece_id=RB2ZE5QKZY2QTRK3BZ3V6F6AVJISFJRQ4MRYUSN6FMA6JZSU6ESA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.787943176 (60h29m32.510506217s) [noise=true , address=104.36.82.52:4091 , piece_id=SLFR5XIA3BYL4G5BKBOKCVOICJUGHJXDXYVXUTMKSZ3ACKPPP3YQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788012329 (60h29m32.510575369s) [noise=true , address=104.36.82.19:4408 , piece_id=YLTM6OBYPAQVZR7W6XXTXWZ2KYRF6BRFGA4NSOZE5G26F4X6TAYA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788241988 (60h29m32.510805028s) [noise=true , address=45.91.116.9:4277 , piece_id=WBXXC2LSJFSLCHHP2YOWNASDKMD7YHOXGLHZPQZ7LA3CTYIKD3NQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788362539 (60h29m32.510925569s) [noise=true , address=79.127.205.246:4258 , piece_id=5FTQPKUVVCBTWWTIDDVF5QHXIWN6N2D5AEY6EQNZVMLQBM3PBRCQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788489312 (60h29m32.511052342s) [noise=true , address=104.36.82.47:4530 , piece_id=HINBU7RQ7ZP2H6Y3IYHYBVTDSJAFMCD26V3EJBBYSTFUCNKFXLVQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788635791 (60h29m32.511198821s) [noise=true , address=79.127.205.234:4607 , piece_id=DLVFNWFBT5VD5K2YEQLHWFXCOZTRYHLVOSOY4XRVNPML5PFXHMDQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788801899 (60h29m32.511364939s) [noise=true , address=104.36.82.46:4511 , piece_id=7ODT6CSGTF57AJORRAKGPKNKQ5NGNUAP75PN3BTLIEFSZ23EFQAQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.788915456 (60h29m32.511478496s) [noise=true , address=77.90.31.10:4523 , piece_id=64GDOWJMFH72AGX7RSPKO5F5QUVZGGVFFFKD5JMZTMBY2F7MITYA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789048411 (60h29m32.511611451s) [noise=true , address=64.49.13.138:4483 , piece_id=X5RAUYAQPVXEP5PPNYOSBBE4Q3SH6ADMSFAQASZ4BN3YUDD6RZ7A, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789151699 (60h29m32.511714739s) [noise=true , address=104.36.82.56:4231 , piece_id=VZXEM3AKXD37SXJDTLSVQXRVKMC2KN22BKKD4JESEP74W4RRIYWQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789284383 (60h29m32.511847423s) [noise=true , address=104.36.82.53:4125 , piece_id=Z2N7PEESJFYSJ2EWAJVD7HN2YNQ6AT5HEGAEHUKYK26QVPW7ACEQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789392951 (60h29m32.511955991s) [noise=true , address=79.127.201.215:4345 , piece_id=OH4NQ72ANUPWVCW6AKEWFPUZFYQNQTQFZF2WCCBCMMYB5W42XDBQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789488373 (60h29m32.512051413s) [noise=true , address=77.90.31.19:4867 , piece_id=4TGQ2CNWNJ6ZBACLLU7HQ44FPK34ERCNQOKA47NYQLRQC53Z5XQA, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789637699 (60h29m32.512200739s) [noise=true , address=79.127.205.235:4657 , piece_id=EJWS4UFR4RRURJXWVXE46SLA5GDH3RFUYZVLFSZKD4BAUVBSRMNQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789684658 (60h29m32.512247688s) [noise=true , address=104.36.82.54:4172 , piece_id=EJNYO5KJ6F3D3JIZQNGN3N7PHL24ABZTYCUNUCZEOUFV6VBX5QBQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:22.789908547 (60h29m32.512471587s) [noise=true , address=79.127.220.101:4041 , piece_id=5TKGOYKYFD37YM2HSOIZ7LJKAQM55DLVW6QAKUBGUJNOZX5RUTPQ, seg_pos={0 0}, upload=58019]: Uploading piece... Jul 27 00:05:23.902236566 (60h29m33.624799596s) [upload=58019]: Next returned nil segment Jul 27 00:05:23.902253699 (60h29m33.624816739s) [upload=58019]: Waiting for error group managing segments... Jul 27 00:05:24.227516616 (60h29m33.950079656s) [noise=true , address=79.127.205.235:4657 , piece_id=EJWS4UFR4RRURJXWVXE46SLA5GDH3RFUYZVLFSZKD4BAUVBSRMNQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:24.522516541 (60h29m34.245079581s) [noise=true , address=79.127.205.246:4258 , piece_id=5FTQPKUVVCBTWWTIDDVF5QHXIWN6N2D5AEY6EQNZVMLQBM3PBRCQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:24.787777739 (60h29m34.510340779s) [noise=true , address=77.90.31.16:4759 , piece_id=OV6NCZRVNDFWVK35MASHEG2UFGQG33H6I2PTJJFLRFVPMSABD5ZA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:24.939622529 (60h29m34.662185569s) [noise=true , address=79.127.205.234:4607 , piece_id=DLVFNWFBT5VD5K2YEQLHWFXCOZTRYHLVOSOY4XRVNPML5PFXHMDQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:27.214858358 (60h29m36.937421398s) [noise=true , address=79.127.220.101:4041 , piece_id=5TKGOYKYFD37YM2HSOIZ7LJKAQM55DLVW6QAKUBGUJNOZX5RUTPQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:28.083413763 (60h29m37.805976803s) [noise=true , address=64.49.13.138:4483 , piece_id=X5RAUYAQPVXEP5PPNYOSBBE4Q3SH6ADMSFAQASZ4BN3YUDD6RZ7A, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:29.775391290 (60h29m39.49795432s) [noise=true , address=45.91.116.9:4277 , piece_id=WBXXC2LSJFSLCHHP2YOWNASDKMD7YHOXGLHZPQZ7LA3CTYIKD3NQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:05:31.705915531 (60h29m41.428478571s) [noise=true , address=77.90.31.19:4867 , piece_id=4TGQ2CNWNJ6ZBACLLU7HQ44FPK34ERCNQOKA47NYQLRQC53Z5XQA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:34.195265036 (60h29m43.917828076s) [noise=true , address=45.91.116.14:4206 , piece_id=RB2ZE5QKZY2QTRK3BZ3V6F6AVJISFJRQ4MRYUSN6FMA6JZSU6ESA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:05:56.693782311 (60h30m6.416345351s) [noise=true , address=79.127.201.215:4345 , piece_id=OH4NQ72ANUPWVCW6AKEWFPUZFYQNQTQFZF2WCCBCMMYB5W42XDBQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: <nil> Jul 27 00:06:21.700033712 (60h30m31.422596782s) [noise=true , address=104.36.82.52:4091 , piece_id=SLFR5XIA3BYL4G5BKBOKCVOICJUGHJXDXYVXUTMKSZ3ACKPPP3YQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.700386977 (60h30m31.422950037s) [noise=true , address=77.90.31.10:4523 , piece_id=64GDOWJMFH72AGX7RSPKO5F5QUVZGGVFFFKD5JMZTMBY2F7MITYA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: read tcp 188.166.225.170:43628->77.90.31.10:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.700918505 (60h30m31.423481545s) [noise=true , address=104.36.82.46:4511 , piece_id=7ODT6CSGTF57AJORRAKGPKNKQ5NGNUAP75PN3BTLIEFSZ23EFQAQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: rpc: dial tcp 104.36.82.46:4511: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.700947921 (60h30m31.423510961s) [noise=true , address=104.36.82.56:4231 , piece_id=VZXEM3AKXD37SXJDTLSVQXRVKMC2KN22BKKD4JESEP74W4RRIYWQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701018105 (60h30m31.423581165s) [noise=true , address=104.36.82.54:4172 , piece_id=EJNYO5KJ6F3D3JIZQNGN3N7PHL24ABZTYCUNUCZEOUFV6VBX5QBQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 188.166.225.170:33396->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701057230 (60h30m31.42362029s) [noise=true , address=104.36.82.19:4408 , piece_id=YLTM6OBYPAQVZR7W6XXTXWZ2KYRF6BRFGA4NSOZE5G26F4X6TAYA, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ, address:104.36.82.19:4408): protocol: rpc: dial tcp 104.36.82.19:4408: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701066748 (60h30m31.423629788s) [noise=true , address=104.36.82.53:4125 , piece_id=Z2N7PEESJFYSJ2EWAJVD7HN2YNQ6AT5HEGAEHUKYK26QVPW7ACEQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:104.36.82.53:4125): protocol: rpc: dial tcp 104.36.82.53:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701071377 (60h30m31.423634417s) [noise=true , address=104.36.82.47:4530 , piece_id=HINBU7RQ7ZP2H6Y3IYHYBVTDSJAFMCD26V3EJBBYSTFUCNKFXLVQ, seg_pos={0 0}, upload=58019]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:21.701094822 (60h30m31.423657922s) [seg_pos={0 0}, upload=58019]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:06:21.944972146 (60h30m31.667535186s) [upload=58019]: Done uploading segments... Jul 27 00:06:21.944996322 (60h30m31.667559362s) [upload=58019]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.246385647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.........] duration: 30.384339509s FAIL (2025-07-27 00:06:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.38435079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.X] duration: 30.23505612s FAIL (2025-07-27 00:05:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235082952s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X........] duration: 38.482893493s FAIL (2025-07-27 00:05:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.482914954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XX] duration: 30.241147367s FAIL (2025-07-27 00:05:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:01.447401116 (60h29m11.169964186s) [seg_pos={0 0}, upload=58008]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:05:01.690687079 (60h29m11.413250129s) [upload=58008]: Done uploading segments... Jul 27 00:05:01.690718899 (60h29m11.413281939s) [upload=58008]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:05:02.560967093 (60h29m12.283530133s) [upload=58013]: Starting upload Jul 27 00:05:02.561059250 (60h29m12.2836223s) [upload=58013]: Done starting upload Jul 27 00:05:02.561208304 (60h29m12.283771364s) [upload=58013]: Uploading segments... Jul 27 00:05:02.561391195 (60h29m12.283954245s) [upload=58013]: Got next segment. Inline: false Jul 27 00:05:02.561445709 (60h29m12.284008749s) [upload=58013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:05:02.561452602 (60h29m12.284015652s) [upload=58013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:05:02.798729347 (60h29m12.521292407s) [seg_pos={0 0}, upload=58013]: Begin upload segment... Jul 27 00:05:02.798805122 (60h29m12.521368162s) [seg_pos={0 0}, upload=58013]: Done begin upload segment. Jul 27 00:05:02.798838295 (60h29m12.521401346s) [noise=true , address=77.90.31.13:4626 , piece_id=2MZ5QDUHJCJZ2CRAJFU3L4RVLIGP32ERMLCR5DAKADWU5MKUQJAA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.798888512 (60h29m12.521451552s) [noise=true , address=77.90.31.15:4699 , piece_id=S3T2ORLRPFLW6ISGDGSHR7U6RPPQIUU2A3EO5AE7SWHFBLCTAX4Q, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.799021236 (60h29m12.521584276s) [noise=true , address=104.36.82.3:4765 , piece_id=HBMIVDDLHPBH3DCBKB4CU4QY5O47FWRKDDR6OKAGCBAYRRT55GVA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.799451228 (60h29m12.522014278s) [noise=true , address=45.91.116.10:4081 , piece_id=H42UNKPAPYBC6Y4GKMMX7MMC44LJ4L5IYRHKSZGTFWVN42QMS3RA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.799464424 (60h29m12.522027464s) [noise=true , address=77.90.31.12:4590 , piece_id=HTERKMF2LZG23VQ3AMBORXNVYOFC65OKVYVKICO5BXP23HZ4LOFA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.799794074 (60h29m12.522357124s) [noise=true , address=45.91.116.7:4011 , piece_id=BWDT7MTYRODWMXJIA3POFLWAPCTSY4HD4A5KQNVA6KAOAVIIS3ZA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.799850472 (60h29m12.522413512s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKORLXGDHVD4D25KFP5QW2ZLPREGY3E5TAQBWS23HZVKUO5I3LQ, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.800176917 (60h29m12.522739967s) [noise=true , address=45.91.116.14:4190 , piece_id=J6QDMYOMJA25SKX2LZE6MAOGMIZGK2SEIHEA6QUW4ONCG5V2PLQA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.800307928 (60h29m12.522870978s) [noise=true , address=77.90.31.14:4663 , piece_id=7K3PMS67INYORWMGTQDJ7QI33F562GARO53AHPOI4FUFOC6EUJBQ, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.800695489 (60h29m12.523258529s) [noise=true , address=104.36.82.12:4150 , piece_id=IQU5VES2L645KCAZFG63GUO6F5SEBQPQM4OH6A2RPRWVJFYVO2UQ, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.801003488 (60h29m12.523566528s) [noise=true , address=77.90.31.10:4518 , piece_id=JBD56FUTX423KXWYUXY4MZ7B3GY6LUMBDQACUR35JJ47X7U2EA7A, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.801329863 (60h29m12.523892913s) [noise=true , address=77.90.31.18:4806 , piece_id=L6CQHIKXKTDX7HKB3CSWSLAAZDDXU5UD522SHAAHBDVFI5RZ2H6Q, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.801678690 (60h29m12.52424174s) [noise=true , address=104.36.82.11:4113 , piece_id=ZGQHVEQGXPLVS3HACWUOSXCLBGSBCRYQ7HIU5PXWJE4WSSYKCCJQ, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.802064338 (60h29m12.524627388s) [noise=true , address=77.90.31.19:4842 , piece_id=4NVG57UEGSRZD3LJFIAIN25YACV6336TARYWSGMDE6FYX4FNZBRQ, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.802411622 (60h29m12.524974672s) [noise=true , address=45.91.116.11:4118 , piece_id=3FV6QW2IXNZASQOCZU6SFIYY3MEFLGQFGQVJZI5ZCS76RJWJGRDA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:02.805333012 (60h29m12.527896052s) [noise=true , address=104.36.82.4:4801 , piece_id=UPUBUUVGCNYNH7QPFL37X4PAEDIM443S63K2AUV6PFFUKSSLX4QA, seg_pos={0 0}, upload=58013]: Uploading piece... Jul 27 00:05:04.611541965 (60h29m14.334105005s) [upload=58013]: Next returned nil segment Jul 27 00:05:04.611564408 (60h29m14.334127448s) [upload=58013]: Waiting for error group managing segments... Jul 27 00:05:04.914940269 (60h29m14.637503309s) [noise=true , address=45.91.116.10:4081 , piece_id=H42UNKPAPYBC6Y4GKMMX7MMC44LJ4L5IYRHKSZGTFWVN42QMS3RA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:04.918629126 (60h29m14.641192166s) [noise=true , address=45.91.116.7:4011 , piece_id=BWDT7MTYRODWMXJIA3POFLWAPCTSY4HD4A5KQNVA6KAOAVIIS3ZA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:04.948737141 (60h29m14.671300181s) [noise=true , address=45.91.116.11:4118 , piece_id=3FV6QW2IXNZASQOCZU6SFIYY3MEFLGQFGQVJZI5ZCS76RJWJGRDA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:04.963852912 (60h29m14.686415952s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKORLXGDHVD4D25KFP5QW2ZLPREGY3E5TAQBWS23HZVKUO5I3LQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:04.988745859 (60h29m14.711308929s) [noise=true , address=77.90.31.14:4663 , piece_id=7K3PMS67INYORWMGTQDJ7QI33F562GARO53AHPOI4FUFOC6EUJBQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:04.991103750 (60h29m14.71366679s) [noise=true , address=45.91.116.14:4190 , piece_id=J6QDMYOMJA25SKX2LZE6MAOGMIZGK2SEIHEA6QUW4ONCG5V2PLQA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:06.248956147 (60h29m15.971519197s) [noise=true , address=77.90.31.18:4806 , piece_id=L6CQHIKXKTDX7HKB3CSWSLAAZDDXU5UD522SHAAHBDVFI5RZ2H6Q, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:06.300358975 (60h29m16.022922016s) [noise=true , address=77.90.31.13:4626 , piece_id=2MZ5QDUHJCJZ2CRAJFU3L4RVLIGP32ERMLCR5DAKADWU5MKUQJAA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:06.410168350 (60h29m16.13273139s) [noise=true , address=77.90.31.12:4590 , piece_id=HTERKMF2LZG23VQ3AMBORXNVYOFC65OKVYVKICO5BXP23HZ4LOFA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:06.869083350 (60h29m16.59164639s) [noise=true , address=77.90.31.19:4842 , piece_id=4NVG57UEGSRZD3LJFIAIN25YACV6336TARYWSGMDE6FYX4FNZBRQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil> Jul 27 00:05:31.913473966 (60h29m41.636036996s) [noise=true , address=104.36.82.4:4801 , piece_id=UPUBUUVGCNYNH7QPFL37X4PAEDIM443S63K2AUV6PFFUKSSLX4QA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:31.913502281 (60h29m41.636065321s) [noise=true , address=77.90.31.10:4518 , piece_id=JBD56FUTX423KXWYUXY4MZ7B3GY6LUMBDQACUR35JJ47X7U2EA7A, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:43888->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:31.913521568 (60h29m41.636084608s) [noise=true , address=104.36.82.3:4765 , piece_id=HBMIVDDLHPBH3DCBKB4CU4QY5O47FWRKDDR6OKAGCBAYRRT55GVA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:56046->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:31.913530945 (60h29m41.636093985s) [noise=true , address=77.90.31.15:4699 , piece_id=S3T2ORLRPFLW6ISGDGSHR7U6RPPQIUU2A3EO5AE7SWHFBLCTAX4Q, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:31.913536205 (60h29m41.636099245s) [noise=true , address=104.36.82.12:4150 , piece_id=IQU5VES2L645KCAZFG63GUO6F5SEBQPQM4OH6A2RPRWVJFYVO2UQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:31.913552858 (60h29m41.636115898s) [noise=true , address=104.36.82.11:4113 , piece_id=ZGQHVEQGXPLVS3HACWUOSXCLBGSBCRYQ7HIU5PXWJE4WSSYKCCJQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:31.913571243 (60h29m41.636134283s) [seg_pos={0 0}, upload=58013]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:05:32.152667560 (60h29m41.8752306s) [upload=58013]: Done uploading segments... Jul 27 00:05:32.152698479 (60h29m41.87526152s) [upload=58013]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.241166312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.65925967s FAIL (2025-07-27 00:05:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.659289076s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229973629s FAIL (2025-07-27 00:05:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== g piece... Jul 27 00:04:09.199649906 (60h28m18.922212946s) [noise=true , address=104.36.82.46:4502 , piece_id=4CRBL5C5EWHPSSGL2CKHW6BSRTOBKGG37TMOA4WOY33GGARTRRNA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.200035684 (60h28m18.922598725s) [noise=true , address=79.127.220.101:4048 , piece_id=OURI6N4Y56D63UICBUEE7QZF4M3EYYEVJ7AX2O5ZE3ND2EY42V2A, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.200040494 (60h28m18.922603534s) [noise=true , address=79.127.205.246:4281 , piece_id=TBFLNGPLPIHQEBWC6QPYBMMGGTR7XMGWRKAQ6QKIUAIONJ4IFGRA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.200375855 (60h28m18.922938895s) [noise=true , address=104.36.82.55:4216 , piece_id=6LBUVHE5KEQHNLAHRHWAMZF3WACHJLGKVTPU7I5GQRFNIDWQ734A, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.200793785 (60h28m18.923356825s) [noise=true , address=79.127.219.51:4210 , piece_id=SA2Q3RCN4I5BADWQLPIFMAXDT5DGRTNO73V4XYNUD3CPSZNXHGYA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.201093718 (60h28m18.923656758s) [noise=true , address=104.36.82.54:4157 , piece_id=DK2ANOFREWW6INIEKDM3JNIW3WLYYF4QMU5OH5CYEMCGWER2RPZQ, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.201396357 (60h28m18.923959397s) [noise=true , address=79.127.205.233:4646 , piece_id=LCSEKEFIVJQ6KEDHCGM36HHCJI7WOKT5ZGBUL6AQY2BP4DTGZMYA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.201691182 (60h28m18.924254222s) [noise=true , address=104.36.82.2:4755 , piece_id=ICLCWXWWQNCEXEIZPMU4EIZ64X7FBDBUTQE2UAB3246FF7O2Q7JQ, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.201979143 (60h28m18.924542183s) [noise=true , address=79.127.205.235:4666 , piece_id=JGZE7SDPWWKSSY2VDFDRF275325EYKVS6NKWSEGC6ESTR5RYV7IA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.202211828 (60h28m18.924774868s) [noise=true , address=79.127.219.47:4048 , piece_id=CX5MVPF4H5Q36CDDBNSLSIQJBXTN3RNP6LJVLN3NFGTJDZVKFQQQ, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.202447349 (60h28m18.925010389s) [noise=true , address=104.36.82.53:4138 , piece_id=YGXKUFVOJ75DZQLVDIBL3ZVNIWO2ZILUPQAA7HMNNBQ47RI3EDMA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.202532802 (60h28m18.925095842s) [noise=true , address=79.127.205.245:4245 , piece_id=NUR2O6WVYPXSEF3ZYZ3JGV2677DGLGKIIJ4W5AYEGI4U6SPEDHAA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.202798570 (60h28m18.92536161s) [noise=true , address=104.36.82.52:4117 , piece_id=CPIV5XMSHAU2WXMGVYKCLKUIKFYTFH4GRU7XECMKZ2VTQ6COR33A, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.203036606 (60h28m18.925599646s) [noise=true , address=79.127.219.38:4010 , piece_id=IJDYW5SSRM2OR3YFRJLLNIWOZSYYIUAXFUS63SW4244WDB2FJOLQ, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.203222462 (60h28m18.925785502s) [noise=true , address=104.36.82.56:4253 , piece_id=MKFS5MKPYG7G4KA3IHLR4S73OZFPOZSFTEQFADPILMSWV7WWM3WA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.203476147 (60h28m18.926039177s) [noise=true , address=104.36.82.7:4918 , piece_id=UDOCYBP77WWFDMTZQWSRXFAEAO3ZK5ZCTSIVNRYE4PTIIMBQBODA, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:09.203826307 (60h28m18.926389337s) [noise=true , address=156.146.43.225:4448 , piece_id=QEF6APTS3KQNEHGSWSL2KLGI7H6MATIYGLCSJ2MC6JT7UKUXB2EQ, seg_pos={0 0}, upload=58000]: Uploading piece... Jul 27 00:04:10.303379963 (60h28m20.025942993s) [upload=58000]: Next returned nil segment Jul 27 00:04:10.303396825 (60h28m20.025959865s) [upload=58000]: Waiting for error group managing segments... Jul 27 00:04:10.468485635 (60h28m20.191048675s) [noise=true , address=79.127.219.51:4210 , piece_id=SA2Q3RCN4I5BADWQLPIFMAXDT5DGRTNO73V4XYNUD3CPSZNXHGYA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 27 00:04:10.618950635 (60h28m20.341513666s) [noise=true , address=79.127.205.246:4281 , piece_id=TBFLNGPLPIHQEBWC6QPYBMMGGTR7XMGWRKAQ6QKIUAIONJ4IFGRA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:10.623707968 (60h28m20.346271008s) [noise=true , address=79.127.205.245:4245 , piece_id=NUR2O6WVYPXSEF3ZYZ3JGV2677DGLGKIIJ4W5AYEGI4U6SPEDHAA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:10.628858482 (60h28m20.351421522s) [noise=true , address=79.127.205.233:4646 , piece_id=LCSEKEFIVJQ6KEDHCGM36HHCJI7WOKT5ZGBUL6AQY2BP4DTGZMYA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:10.632026083 (60h28m20.354589123s) [noise=true , address=79.127.205.235:4666 , piece_id=JGZE7SDPWWKSSY2VDFDRF275325EYKVS6NKWSEGC6ESTR5RYV7IA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:11.122563874 (60h28m20.845126914s) [noise=true , address=79.127.219.38:4010 , piece_id=IJDYW5SSRM2OR3YFRJLLNIWOZSYYIUAXFUS63SW4244WDB2FJOLQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:11.445377285 (60h28m21.167940325s) [noise=true , address=79.127.220.101:4048 , piece_id=OURI6N4Y56D63UICBUEE7QZF4M3EYYEVJ7AX2O5ZE3ND2EY42V2A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:12.775062857 (60h28m22.497625897s) [noise=true , address=79.127.219.47:4048 , piece_id=CX5MVPF4H5Q36CDDBNSLSIQJBXTN3RNP6LJVLN3NFGTJDZVKFQQQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:12.926457887 (60h28m22.649020927s) [noise=true , address=156.146.43.225:4448 , piece_id=QEF6APTS3KQNEHGSWSL2KLGI7H6MATIYGLCSJ2MC6JT7UKUXB2EQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:04:21.769452468 (60h28m31.492015508s) [noise=true , address=64.49.13.138:4488 , piece_id=5XRHQYWD2BJXGHH7ABFJXDDXIVUS63LRSGRT2SXITYWSJ2WMP2FA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil> Jul 27 00:05:08.019334096 (60h29m17.741897136s) [noise=true , address=104.36.82.56:4253 , piece_id=MKFS5MKPYG7G4KA3IHLR4S73OZFPOZSFTEQFADPILMSWV7WWM3WA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019386065 (60h29m17.741949105s) [noise=true , address=104.36.82.46:4502 , piece_id=4CRBL5C5EWHPSSGL2CKHW6BSRTOBKGG37TMOA4WOY33GGARTRRNA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019428085 (60h29m17.741991125s) [noise=true , address=104.36.82.53:4138 , piece_id=YGXKUFVOJ75DZQLVDIBL3ZVNIWO2ZILUPQAA7HMNNBQ47RI3EDMA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019461268 (60h29m17.742024309s) [noise=true , address=104.36.82.52:4117 , piece_id=CPIV5XMSHAU2WXMGVYKCLKUIKFYTFH4GRU7XECMKZ2VTQ6COR33A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:104.36.82.52:4117): protocol: rpc: dial tcp 104.36.82.52:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019499592 (60h29m17.742062632s) [noise=true , address=104.36.82.55:4216 , piece_id=6LBUVHE5KEQHNLAHRHWAMZF3WACHJLGKVTPU7I5GQRFNIDWQ734A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019535661 (60h29m17.742098701s) [noise=true , address=104.36.82.7:4918 , piece_id=UDOCYBP77WWFDMTZQWSRXFAEAO3ZK5ZCTSIVNRYE4PTIIMBQBODA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: rpc: dial tcp 104.36.82.7:4918: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019575157 (60h29m17.742138197s) [noise=true , address=104.36.82.54:4157 , piece_id=DK2ANOFREWW6INIEKDM3JNIW3WLYYF4QMU5OH5CYEMCGWER2RPZQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019629892 (60h29m17.742192932s) [noise=true , address=104.36.82.2:4755 , piece_id=ICLCWXWWQNCEXEIZPMU4EIZ64X7FBDBUTQE2UAB3246FF7O2Q7JQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 188.166.225.170:40334->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:08.019646233 (60h29m17.742209273s) [seg_pos={0 0}, upload=58000]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:05:08.246996305 (60h29m17.969559335s) [upload=58000]: Done uploading segments... Jul 27 00:05:08.247022976 (60h29m17.969586016s) [upload=58000]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.229993076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.] duration: 5.944056939s FAIL (2025-07-27 00:05:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.944068381s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.X] duration: 30.245112591s FAIL (2025-07-27 00:05:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UZi3): protocol: expected piece hash; context canceled; EOF Jul 27 00:03:40.824444038 (60h27m50.547007068s) [seg_pos={0 0}, upload=57993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:03:40.824514603 (60h27m50.547077643s) [upload=57993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:03:40.824519062 (60h27m50.547082092s) [upload=57993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:03:41.130112176 (60h27m50.852675216s) [upload=57993]: Done uploading segments... Jul 27 00:03:41.130131623 (60h27m50.852694663s) [upload=57993]: Upload finished. err: <nil> Jul 27 00:04:32.085896910 (60h28m41.80845995s) [upload=58008]: Starting upload Jul 27 00:04:32.085951976 (60h28m41.808515016s) [upload=58008]: Done starting upload Jul 27 00:04:32.086013413 (60h28m41.808576493s) [upload=58008]: Uploading segments... Jul 27 00:04:32.086132050 (60h28m41.80869509s) [upload=58008]: Got next segment. Inline: false Jul 27 00:04:32.086163139 (60h28m41.808726179s) [upload=58008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:04:32.086168039 (60h28m41.808731069s) [upload=58008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:04:32.316154546 (60h28m42.038717586s) [seg_pos={0 0}, upload=58008]: Begin upload segment... Jul 27 00:04:32.316190194 (60h28m42.038753234s) [seg_pos={0 0}, upload=58008]: Done begin upload segment. Jul 27 00:04:32.316211956 (60h28m42.038774986s) [noise=true , address=104.36.82.10:4079 , piece_id=QCH7JNNFNHZMXBGSYIXW7SUQ2VOKR6IXLYJN5EWP5P6Q33EXYPYA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.316550994 (60h28m42.039114034s) [noise=true , address=77.90.31.21:4877 , piece_id=6RDEYAV6BOBL7S5MOV4PFU7CQ7RY5E5BK2Z5RLNPD5WRAJYMP7PQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.316820280 (60h28m42.03938332s) [noise=true , address=45.91.116.10:4083 , piece_id=CMAZSSQVNGW7YLYKDSLWAC7WT4F6CWQU2VWYQHMRPYSP2DTC7NXQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.317062914 (60h28m42.039625954s) [noise=true , address=77.90.31.10:4517 , piece_id=F62P5KID32MAKPMCBACN4R62AABMSJCKP6Z6HG4673XJFIWOR63Q, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.317476846 (60h28m42.040039886s) [noise=true , address=45.91.116.6:4974 , piece_id=7TGQV3NUTHXHPK4D2ATCHS5RYPMO2ZAIBSACLJVMRQHEEOUFEQSA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.317739519 (60h28m42.040302559s) [noise=true , address=77.90.31.14:4661 , piece_id=A47OUBHKNVLGCIQR6ASCDVXMFVN2YHEYX5OXSF5E5MMS6VQSMKKQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.318258591 (60h28m42.040821631s) [noise=true , address=77.90.31.16:4733 , piece_id=MWTJT7ZR36IJRKQ6EDB5M55HEW4MTXXD7K324XOLOSD4TQSKNQQA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.318665460 (60h28m42.0412285s) [noise=true , address=45.91.116.4:4938 , piece_id=XGGUREMQB7IUS2REQGFMSOFZ7XVTVLXPXILMDD3S2MS6N7L4CXEA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.318967077 (60h28m42.041530117s) [noise=true , address=77.90.31.16:4734 , piece_id=ZGQYC7TZ6S4DHGXRUXUPM2JWW5TXFJPBP7ZCCRQVMFURWZKXJTTA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.319263524 (60h28m42.041826565s) [noise=true , address=45.91.116.8:4047 , piece_id=43QUJDR6ZR7676AD5M6JNL4SLVK6MF6HRT5GX5D7PF2INO6GEKUQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.319569590 (60h28m42.04213263s) [noise=true , address=77.90.31.13:4626 , piece_id=H3WZA5CJYXQWWKTKGO7GC7TAHHX43RT4KXFVNDAPN26444VZLFVA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.319889312 (60h28m42.042452352s) [noise=true , address=104.36.82.9:4006 , piece_id=B3ZHSA6BUUCS6KG34JR24HTPUL3EFOD3LK3ZENMMMRCJC7BH3ZXA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.320214715 (60h28m42.042777755s) [noise=true , address=77.90.31.11:4555 , piece_id=GV7OJKDR3H45NKLKWJH2FIMFQ6IZ6XBXTPFFKJSOOI2YX7QF7EEQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.320498017 (60h28m42.043061057s) [noise=true , address=77.90.31.15:4699 , piece_id=LLXRJR5MUYCIT6UB2A3NEQRVXNGTHLAUODC52QZR43COIRZFPULQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.320812748 (60h28m42.043375788s) [noise=true , address=45.91.116.13:4155 , piece_id=O3KD6DANCUFYSTYEDD5ZFUYQZDR3MAUHESK5OXAUZB7EZBPCEAEA, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:32.321152088 (60h28m42.043715138s) [noise=true , address=104.36.82.15:4257 , piece_id=C2VF7M7SKHY5ISTRDAXXKITAHHZZG32QDO5XQDXFQ4VHADMXCPBQ, seg_pos={0 0}, upload=58008]: Uploading piece... Jul 27 00:04:34.057760807 (60h28m43.780323847s) [upload=58008]: Next returned nil segment Jul 27 00:04:34.057779792 (60h28m43.780342832s) [upload=58008]: Waiting for error group managing segments... Jul 27 00:04:34.335101806 (60h28m44.057664846s) [noise=true , address=45.91.116.13:4155 , piece_id=O3KD6DANCUFYSTYEDD5ZFUYQZDR3MAUHESK5OXAUZB7EZBPCEAEA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.339370523 (60h28m44.061933563s) [noise=true , address=77.90.31.14:4661 , piece_id=A47OUBHKNVLGCIQR6ASCDVXMFVN2YHEYX5OXSF5E5MMS6VQSMKKQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.349741225 (60h28m44.072304255s) [noise=true , address=45.91.116.4:4938 , piece_id=XGGUREMQB7IUS2REQGFMSOFZ7XVTVLXPXILMDD3S2MS6N7L4CXEA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.430079845 (60h28m44.152642875s) [noise=true , address=77.90.31.16:4734 , piece_id=ZGQYC7TZ6S4DHGXRUXUPM2JWW5TXFJPBP7ZCCRQVMFURWZKXJTTA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.475555101 (60h28m44.198118141s) [noise=true , address=77.90.31.21:4877 , piece_id=6RDEYAV6BOBL7S5MOV4PFU7CQ7RY5E5BK2Z5RLNPD5WRAJYMP7PQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.479221716 (60h28m44.201784746s) [noise=true , address=45.91.116.6:4974 , piece_id=7TGQV3NUTHXHPK4D2ATCHS5RYPMO2ZAIBSACLJVMRQHEEOUFEQSA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.497190286 (60h28m44.219753326s) [noise=true , address=77.90.31.11:4555 , piece_id=GV7OJKDR3H45NKLKWJH2FIMFQ6IZ6XBXTPFFKJSOOI2YX7QF7EEQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.499780581 (60h28m44.222343621s) [noise=true , address=45.91.116.10:4083 , piece_id=CMAZSSQVNGW7YLYKDSLWAC7WT4F6CWQU2VWYQHMRPYSP2DTC7NXQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.502073518 (60h28m44.224636558s) [noise=true , address=77.90.31.16:4733 , piece_id=MWTJT7ZR36IJRKQ6EDB5M55HEW4MTXXD7K324XOLOSD4TQSKNQQA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:34.562247958 (60h28m44.284810988s) [noise=true , address=45.91.116.8:4047 , piece_id=43QUJDR6ZR7676AD5M6JNL4SLVK6MF6HRT5GX5D7PF2INO6GEKUQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:04:36.427408290 (60h28m46.14997133s) [noise=true , address=77.90.31.13:4626 , piece_id=H3WZA5CJYXQWWKTKGO7GC7TAHHX43RT4KXFVNDAPN26444VZLFVA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil> Jul 27 00:05:01.447187046 (60h29m11.169750086s) [noise=true , address=104.36.82.15:4257 , piece_id=C2VF7M7SKHY5ISTRDAXXKITAHHZZG32QDO5XQDXFQ4VHADMXCPBQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:56162->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:01.447238865 (60h29m11.169801905s) [noise=true , address=77.90.31.15:4699 , piece_id=LLXRJR5MUYCIT6UB2A3NEQRVXNGTHLAUODC52QZR43COIRZFPULQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:01.447268642 (60h29m11.169831682s) [noise=true , address=104.36.82.10:4079 , piece_id=QCH7JNNFNHZMXBGSYIXW7SUQ2VOKR6IXLYJN5EWP5P6Q33EXYPYA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:01.447297848 (60h29m11.169860898s) [noise=true , address=104.36.82.9:4006 , piece_id=B3ZHSA6BUUCS6KG34JR24HTPUL3EFOD3LK3ZENMMMRCJC7BH3ZXA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:42962->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:01.447363344 (60h29m11.169926394s) [noise=true , address=77.90.31.10:4517 , piece_id=F62P5KID32MAKPMCBACN4R62AABMSJCKP6Z6HG4673XJFIWOR63Q, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58900->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:01.447401116 (60h29m11.169964186s) [seg_pos={0 0}, upload=58008]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:05:01.690687079 (60h29m11.413250129s) [upload=58008]: Done uploading segments... Jul 27 00:05:01.690718899 (60h29m11.413281939s) [upload=58008]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.245134211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X] duration: 30.001229295s FAIL (2025-07-27 00:04:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001246707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.] duration: 7.109247136s FAIL (2025-07-27 00:04:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.109270831s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.854811833s FAIL (2025-07-27 00:04:30 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.854880083s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX] duration: 30.227430826s FAIL (2025-07-27 00:04:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.227452569s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.......] duration: 10.157508953s FAIL (2025-07-27 00:04:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.157530234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234010391s FAIL (2025-07-27 00:04:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 429914s) [noise=true , address=45.91.116.14:4206 , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296296305 (60h27m14.018859355s) [noise=true , address=104.36.82.40:4962 , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296448477 (60h27m14.019011567s) [noise=true , address=79.127.205.233:4633 , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296801893 (60h27m14.019364923s) [noise=true , address=79.127.219.38:4009 , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.296916422 (60h27m14.019479462s) [noise=true , address=64.49.13.137:4502 , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297216586 (60h27m14.019779626s) [noise=true , address=79.127.205.234:4611 , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297386952 (60h27m14.019949992s) [noise=true , address=64.49.13.154:4927 , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297594509 (60h27m14.020157549s) [noise=true , address=77.90.31.10:4543 , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297733676 (60h27m14.020296716s) [noise=true , address=104.36.82.46:4504 , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.297961742 (60h27m14.020524782s) [noise=true , address=104.36.82.54:4171 , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298019222 (60h27m14.020582262s) [noise=true , address=77.90.31.12:4620 , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298239854 (60h27m14.020802894s) [noise=true , address=79.127.205.245:4252 , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298338854 (60h27m14.020901894s) [noise=true , address=79.127.205.235:4669 , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298502326 (60h27m14.021065366s) [noise=true , address=104.36.82.47:4552 , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298706026 (60h27m14.021269066s) [noise=true , address=104.36.82.14:4245 , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.298868206 (60h27m14.021431246s) [noise=true , address=79.127.201.215:4347 , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.299167178 (60h27m14.021730229s) [noise=true , address=104.36.82.56:4230 , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:04.299589586 (60h27m14.022152626s) [noise=true , address=104.36.82.12:4169 , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Uploading piece... Jul 27 00:03:05.385782620 (60h27m15.10834566s) [upload=57985]: Next returned nil segment Jul 27 00:03:05.385828919 (60h27m15.108391969s) [upload=57985]: Waiting for error group managing segments... Jul 27 00:03:05.701031808 (60h27m15.423594848s) [noise=true , address=79.127.205.234:4611 , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:05.701960976 (60h27m15.424524006s) [noise=true , address=79.127.205.235:4669 , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:05.722094938 (60h27m15.444657968s) [noise=true , address=79.127.205.245:4252 , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:05.743552713 (60h27m15.466115783s) [noise=true , address=79.127.205.233:4633 , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:06.278512358 (60h27m16.001075398s) [noise=true , address=79.127.219.38:4009 , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:08.340018352 (60h27m18.062581392s) [noise=true , address=77.90.31.12:4620 , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:11.565721846 (60h27m21.288284886s) [noise=true , address=64.49.13.137:4502 , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:13.988632217 (60h27m23.711195257s) [noise=true , address=64.49.13.154:4927 , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:18.683164753 (60h27m28.405727793s) [noise=true , address=45.91.116.14:4206 , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:03:30.503118221 (60h27m40.225681261s) [noise=true , address=79.127.201.215:4347 , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil> Jul 27 00:04:03.190575544 (60h28m12.913138584s) [noise=true , address=104.36.82.54:4171 , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190656027 (60h28m12.913219067s) [noise=true , address=104.36.82.12:4169 , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190692197 (60h28m12.913255237s) [noise=true , address=104.36.82.56:4230 , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190745179 (60h28m12.913308219s) [noise=true , address=104.36.82.46:4504 , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:37536->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190772331 (60h28m12.913335371s) [noise=true , address=104.36.82.40:4962 , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:104.36.82.40:4962): protocol: rpc: dial tcp 104.36.82.40:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190827717 (60h28m12.913390757s) [noise=true , address=77.90.31.10:4543 , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190887290 (60h28m12.91345033s) [noise=true , address=104.36.82.47:4552 , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190940382 (60h28m12.913503432s) [noise=true , address=104.36.82.14:4245 , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:03.190980088 (60h28m12.913543158s) [seg_pos={0 0}, upload=57985]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:04:03.422740258 (60h28m13.145303289s) [upload=57985]: Done uploading segments... Jul 27 00:04:03.422770667 (60h28m13.145333717s) [upload=57985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.234036131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X......] duration: 32.74714383s FAIL (2025-07-27 00:03:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.747160172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......] duration: 6.248021118s FAIL (2025-07-27 00:03:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.248043411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.] duration: 5.808010915s FAIL (2025-07-27 00:03:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments... Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments... Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment... Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment. Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments... Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments... Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.808034329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.842571085s FAIL (2025-07-27 00:03:33 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.842596432s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 30.237202557s FAIL (2025-07-27 00:03:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments... Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments... Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment... Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment. Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece... Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments... Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil> Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments... Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.237224419s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.....] duration: 50.911700599s FAIL (2025-07-27 00:03:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.911720456s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)