Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 11.966786184s FAIL (2025-07-28 01:37:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 11.966820019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.....XXX.......X....X.....X......X.........] duration: 5.712871223s FAIL (2025-07-28 01:36:57 UTC) orbiter warning: uplink: too many requests Duration: 5.712963199s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 7.459160322s FAIL (2025-07-28 01:36:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.459178966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.........X......X..X.....................] duration: 5.80974867s FAIL (2025-07-28 01:36:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.809777145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..................X.........X......X..X....................] duration: 5.825940427s FAIL (2025-07-28 01:35:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.825963492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.....XXX.......X....X.....X......X........] duration: 5.711890487s FAIL (2025-07-28 01:35:46 UTC) orbiter warning: uplink: too many requests Duration: 5.711910676s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..................X.........X......X..X...................] duration: 5.712717429s FAIL (2025-07-28 01:35:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712741485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration: 11.935620746s FAIL (2025-07-28 01:35:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 11.935653177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..................X.........X......X..X..................] duration: 5.832075976s FAIL (2025-07-28 01:35:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832110882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration: 13.415029714s FAIL (2025-07-28 01:34:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 13.415056384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.....XXX.......X....X.....X......X.......] duration: 5.876787345s FAIL (2025-07-28 01:34:56 UTC) orbiter warning: uplink: too many requests Duration: 5.876814366s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 11.365416416s FAIL (2025-07-28 01:34:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 11.365441294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..................X.........X......X..X.................] duration: 5.775958779s FAIL (2025-07-28 01:34:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775987103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 15.812488479s FAIL (2025-07-28 01:34:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 15.812514439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.....XXX.......X....X.....X......X......] duration: 5.848248167s FAIL (2025-07-28 01:33:52 UTC) orbiter warning: uplink: too many requests Duration: 5.848274418s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..................X.........X......X..X................] duration: 5.785344176s FAIL (2025-07-28 01:33:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.785373021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..................X.........X......X..X...............] duration: 5.785910139s FAIL (2025-07-28 01:33:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.785933583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X.....XXX.......X....X.....X......X.....] duration: 5.683551599s FAIL (2025-07-28 01:33:17 UTC) orbiter warning: uplink: too many requests Duration: 5.683585544s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 13.016532432s FAIL (2025-07-28 01:32:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 13.016562699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..................X.........X......X..X..............] duration: 6.021013692s FAIL (2025-07-28 01:32:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.021055341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X.....XXX.......X....X.....X......X....] duration: 5.938263066s FAIL (2025-07-28 01:32:50 UTC) orbiter warning: uplink: too many requests Duration: 5.938288083s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..................X.........X......X..X.............] duration: 5.894015376s FAIL (2025-07-28 01:32:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.89404343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 13.692611819s FAIL (2025-07-28 01:32:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 13.692637349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X.....XXX.......X....X.....X......X...] duration: 6.421331187s FAIL (2025-07-28 01:31:47 UTC) orbiter warning: uplink: too many requests Duration: 6.421356035s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X.....XXX.......X....X.....X......X..] duration: 5.799740451s FAIL (2025-07-28 01:31:23 UTC) orbiter warning: uplink: too many requests Duration: 5.799770299s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..................X.........X......X..X............] duration: 5.794479574s FAIL (2025-07-28 01:31:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.794507227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 12.90682171s FAIL (2025-07-28 01:31:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 12.906849262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X.....XXX.......X....X.....X......X.] duration: 6.056160097s FAIL (2025-07-28 01:30:53 UTC) orbiter warning: uplink: too many requests Duration: 6.056207818s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 12.329492923s FAIL (2025-07-28 01:30:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 12.329516729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................X.....XXX.......X....X.....X......X] duration: 879.678693ms FAIL (2025-07-28 01:30:21 UTC) orbiter warning: uplink: too many requests Duration: 879.699794ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..................X.........X......X..X...........] duration: 5.774878053s FAIL (2025-07-28 01:30:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.774913431s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 9.999557821s FAIL (2025-07-28 01:29:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.999582818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..................X.........X......X..X..........] duration: 6.03360337s FAIL (2025-07-28 01:29:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.033630893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 34.633727083s FAIL (2025-07-28 01:29:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 34.633742943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X.....XXX.......X....X.....X......] duration: 5.66408774s FAIL (2025-07-28 01:29:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.664110242s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X..................X.........X......X..X.........] duration: 6.19548191s FAIL (2025-07-28 01:29:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.195541744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X.....XXX.......X....X.....X.....] duration: 5.907218369s FAIL (2025-07-28 01:28:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.907243558s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237557897s FAIL (2025-07-28 01:28:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.237569159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X..................X.........X......X..X........] duration: 5.756538895s FAIL (2025-07-28 01:28:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.756563481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X.....XXX.......X....X.....X....] duration: 6.00378983s FAIL (2025-07-28 01:28:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 6.003817021s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.242154418s FAIL (2025-07-28 01:27:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 608325s) [noise=true , address=104.36.82.22:4538 , piece_id=CAS4FYLJQTQWVKB42WMTO27FU4TPDTV3KOVH5QEX5DTOAFOQASTQ, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486064342 (85h51m4.208627382s) [noise=true , address=104.36.82.6:4908 , piece_id=423GN73JTCL36RLLSO6FSXSWK6HCPZBLL4M76OC257VTGZ6OYJEQ, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486268182 (85h51m4.208831212s) [noise=true , address=104.36.82.55:4220 , piece_id=L7ZHHRUCACBNRWDZRJKRIV4TPOSUWH75L77LOOLNTCATN7DXCZOQ, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486276067 (85h51m4.208839107s) [noise=true , address=104.36.82.11:4139 , piece_id=64LU33OCA7YF7ILTXUKGH3U3G4KJ55VVVJ5RMGTDM2MELDPZN3WA, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486472653 (85h51m4.209035693s) [noise=true , address=104.36.82.52:4113 , piece_id=ML7SOMJWQZHWC2YVYLSRSWH6DITYKWWFEMTVARGLXQ3UZCKRH4SQ, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486486730 (85h51m4.20904977s) [noise=true , address=104.36.82.20:4456 , piece_id=QDL3B6EZ3C4RD5K2BRUR7DG5F7IDHPD57MO7VKACXOXRL5ZVWIBQ, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486674800 (85h51m4.20923784s) [noise=true , address=104.36.82.56:4258 , piece_id=M4K3HL4DLOHMXDLDDXQS56625C7VANVMOKLJWWBG5JN2Y6LPYJFQ, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.486703875 (85h51m4.209266915s) [noise=true , address=104.36.82.6:4894 , piece_id=BHBMTWNSV35VVTJJW5G7ILMBXU7S7XFJJF54R6EZCU4AD4PIYC3Q, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:54.487465954 (85h51m4.210028994s) [noise=true , address=104.36.82.15:4270 , piece_id=XD3TX7IGS7NZ5CU5Z5J4JVITAOORJPJKIITHFZCJJEFY2XU5CHUA, seg_pos={0 0}, upload=82359]: Uploading piece... Jul 28 01:26:55.600183358 (85h51m5.322746398s) [upload=82359]: Next returned nil segment Jul 28 01:26:55.600196523 (85h51m5.322759563s) [upload=82359]: Waiting for error group managing segments... Jul 28 01:26:56.038103908 (85h51m5.760666948s) [noise=true , address=79.127.205.245:4251 , piece_id=PGPHOM3FY7KRXMUOJERCJ5HECNXD5EZHASD5VTVQ52CBASJDL3IQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: <nil> Jul 28 01:26:56.592445486 (85h51m6.315008516s) [noise=true , address=45.91.116.9:4277 , piece_id=IOQRGC775H7TIC66JNNEYGRWT5626VORDSCGQTEZ4R76WGP5QUAA, seg_pos={0 0}, upload=82359]: Done uploading piece. err: <nil> Jul 28 01:26:57.006044772 (85h51m6.728607812s) [noise=true , address=79.127.201.215:4348 , piece_id=HEACLKAIVGNAQMALRUKXRB6TNBE32JSHZRCLDSTLMOPCYZCOWIAA, seg_pos={0 0}, upload=82359]: Done uploading piece. err: <nil> Jul 28 01:27:53.322946444 (85h52m3.045509484s) [noise=true , address=104.36.82.13:4215 , piece_id=U7RXGT2D6L5IDASUTUHZUJSQIJYOVRGI2WIT7FW3D7AFZUJOT7ZQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1vV1hGhZqeEoh5ei7yRu49oyAvTZ2T7S1x3ooufJPtFH2TGBkQ, address:104.36.82.13:4215): protocol: write tcp 188.166.225.170:53552->104.36.82.13:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.322992833 (85h52m3.045555883s) [noise=true , address=104.36.82.6:4908 , piece_id=423GN73JTCL36RLLSO6FSXSWK6HCPZBLL4M76OC257VTGZ6OYJEQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:104.36.82.6:4908): protocol: write tcp 188.166.225.170:50042->104.36.82.6:4908: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323028501 (85h52m3.045591541s) [noise=true , address=104.36.82.11:4139 , piece_id=64LU33OCA7YF7ILTXUKGH3U3G4KJ55VVVJ5RMGTDM2MELDPZN3WA, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: write tcp 188.166.225.170:55778->104.36.82.11:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323058438 (85h52m3.045621478s) [noise=true , address=104.36.82.10:4088 , piece_id=XZNOGB64IAK4P4VLAGWGN7V4IAJTWYAQL3WEVPHGEU2G7OFBM3NA, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, address:104.36.82.10:4088): protocol: write tcp 188.166.225.170:46416->104.36.82.10:4088: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323089418 (85h52m3.045652458s) [noise=true , address=104.36.82.22:4538 , piece_id=CAS4FYLJQTQWVKB42WMTO27FU4TPDTV3KOVH5QEX5DTOAFOQASTQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:104.36.82.22:4538): protocol: write tcp 188.166.225.170:53076->104.36.82.22:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323120167 (85h52m3.045683207s) [noise=true , address=104.36.82.20:4456 , piece_id=QDL3B6EZ3C4RD5K2BRUR7DG5F7IDHPD57MO7VKACXOXRL5ZVWIBQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:104.36.82.20:4456): protocol: write tcp 188.166.225.170:51018->104.36.82.20:4456: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323147048 (85h52m3.045710088s) [noise=true , address=104.36.82.15:4270 , piece_id=XD3TX7IGS7NZ5CU5Z5J4JVITAOORJPJKIITHFZCJJEFY2XU5CHUA, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: write tcp 188.166.225.170:35052->104.36.82.15:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323175282 (85h52m3.045738322s) [noise=true , address=104.36.82.55:4220 , piece_id=L7ZHHRUCACBNRWDZRJKRIV4TPOSUWH75L77LOOLNTCATN7DXCZOQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 188.166.225.170:57828->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323201753 (85h52m3.045764783s) [noise=true , address=104.36.82.52:4113 , piece_id=ML7SOMJWQZHWC2YVYLSRSWH6DITYKWWFEMTVARGLXQ3UZCKRH4SQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou, address:104.36.82.52:4113): protocol: rpc: dial tcp 104.36.82.52:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323290001 (85h52m3.045853041s) [noise=true , address=104.36.82.56:4258 , piece_id=M4K3HL4DLOHMXDLDDXQS56625C7VANVMOKLJWWBG5JN2Y6LPYJFQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: rpc: dial tcp 104.36.82.56:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323326341 (85h52m3.045889381s) [noise=true , address=104.36.82.6:4894 , piece_id=BHBMTWNSV35VVTJJW5G7ILMBXU7S7XFJJF54R6EZCU4AD4PIYC3Q, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:12fZsj77XZ8CnYD6iwzEVKPn6yLyxWMtibzsRXcuGKwNxEXB6Hq, address:104.36.82.6:4894): protocol: rpc: dial tcp 104.36.82.6:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323354134 (85h52m3.045917174s) [noise=true , address=104.36.82.50:4018 , piece_id=24TCJ22VUXJLFXMWMFLZDU7A3K2AH57NLKAQHPPNUDHXT72NQHDQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:12YcYgDWq2k7F9fwpdLAGNkrskGEUURe6QuRed7mRjMTdNHEQ8D, address:104.36.82.50:4018): protocol: rpc: dial tcp 104.36.82.50:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323381326 (85h52m3.045944366s) [noise=true , address=104.36.82.12:4164 , piece_id=3UYKX5PN4K7JK6IX4ODSD26QY6LVHT44QRTIHLTR3VDRHF5TTHAQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: rpc: dial tcp 104.36.82.12:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323422555 (85h52m3.045985605s) [noise=true , address=104.36.82.54:4169 , piece_id=AWEFKOENMK64TEEECZ6DZVJEXPWYU4V5SRYX27OXSHDT2IVYCALQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323446140 (85h52m3.04600918s) [noise=true , address=104.36.82.9:4036 , piece_id=3YKXMQXT4FSXA7DHXQYGQ7FU57POTOFP4XMWCK6FXXELTGKWLHVQ, seg_pos={0 0}, upload=82359]: Done uploading piece. err: ecclient: upload failed (node:12Tcd5VLN93v4Aijkh1Mkyjuu42vKpN93qSGzbn7UGDVNsPAMKy, address:104.36.82.9:4036): protocol: write tcp 188.166.225.170:43308->104.36.82.9:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:53.323463403 (85h52m3.046026443s) [seg_pos={0 0}, upload=82359]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 01:27:53.563311632 (85h52m3.285874673s) [upload=82359]: Done uploading segments... Jul 28 01:27:53.563344665 (85h52m3.285907706s) [upload=82359]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.242167083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X..................X.........X......X..X.......] duration: 5.802135051s FAIL (2025-07-28 01:27:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.802158415s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X.....XXX.......X....X.....X...] duration: 5.846513344s FAIL (2025-07-28 01:27:11 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.846544193s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X..................X.........X......X..X......] duration: 5.880468629s FAIL (2025-07-28 01:26:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.880498276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234394314s FAIL (2025-07-28 01:26:53 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== TVJSFQGVHLBWSEN3VBJD2G2RTYCA, seg_pos={0 0}, upload=82342]: Uploading piece... Jul 28 01:25:53.987335598 (85h50m3.709898638s) [noise=true , address=104.36.82.6:4899 , piece_id=J5CMSTDHBGSLDQG2DK6B5E42CONDZBARTS4HB5LWZFXD47CAW2CQ, seg_pos={0 0}, upload=82342]: Uploading piece... Jul 28 01:25:53.987579114 (85h50m3.710142154s) [noise=true , address=104.36.82.41:4778 , piece_id=S5TUTBLCFYC7FSXRFW7WGWX5W6SY7PWYP2UJVQRGRFKIESPGEQWQ, seg_pos={0 0}, upload=82342]: Uploading piece... Jul 28 01:25:53.987805938 (85h50m3.710368978s) [noise=true , address=104.36.82.54:4158 , piece_id=655JUTMWI47DELEN3AK4AGTWBLPGR3DW4SR7KEVG2LJV5KJ23GQQ, seg_pos={0 0}, upload=82342]: Uploading piece... Jul 28 01:25:53.988012464 (85h50m3.710575504s) [noise=true , address=104.36.82.40:4950 , piece_id=K65YR4JNPKHF5IHQLZELDNDN7XLE3M6BFBPW6BYFT57PKJZIIJIQ, seg_pos={0 0}, upload=82342]: Uploading piece... Jul 28 01:25:53.988202958 (85h50m3.710765998s) [noise=true , address=104.36.82.47:4525 , piece_id=RSKQKU2HSDGA53DE7K5XMA755P6VV6BVUGF7EKKXZL67MP73ZXEA, seg_pos={0 0}, upload=82342]: Uploading piece... Jul 28 01:26:52.865080606 (85h51m2.587643646s) [noise=true , address=104.36.82.51:4083 , piece_id=QQUWJGACQJNL45H5ZEI5QXIMV2THN7HNB6TW2FJ7IAP7P7XY3LXQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1TvsWxZ7L9KYVEZK7nkT14c9GXDcCYCwZQJBcLUfBEqgrufNhZ, address:104.36.82.51:4083): protocol: write tcp 188.166.225.170:33910->104.36.82.51:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865366553 (85h51m2.587929593s) [upload=82342]: Next segment err: context deadline exceeded Jul 28 01:26:52.865443740 (85h51m2.58800678s) [noise=true , address=104.36.82.13:4194 , piece_id=PAUNI6I7HSH7RRPRGEDDIQZRIELSX6S5UX6N5SBXP2HEYR6UREGQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: write tcp 188.166.225.170:42840->104.36.82.13:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865480340 (85h51m2.58804338s) [noise=true , address=104.36.82.47:4525 , piece_id=RSKQKU2HSDGA53DE7K5XMA755P6VV6BVUGF7EKKXZL67MP73ZXEA, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 188.166.225.170:47370->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865528362 (85h51m2.588091402s) [noise=true , address=104.36.82.14:4242 , piece_id=H2EX6IVZK7YWZD7JQGG2ZEHRQFXMF4GUFSEB7R6YMNAU4ZXC5HXQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: write tcp 188.166.225.170:48654->104.36.82.14:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865561165 (85h51m2.588124195s) [noise=true , address=104.36.82.7:4914 , piece_id=IVM3ETQ7IPPKUXGX7AJX3N3QRXKK7QQNSHWVSPXZ66OXJCR6NVJQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1MArPyMrdvyw4VFuArmattTtneNTaN5mt82nfGwY2sKHvQnL5D, address:104.36.82.7:4914): protocol: write tcp 188.166.225.170:32882->104.36.82.7:4914: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865607343 (85h51m2.588170384s) [noise=true , address=104.36.82.5:4859 , piece_id=RE77LAHYVHWMGK4NNEHURQ34XO7OSBOLDGMPKKWIV2LUPJSQKWPQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: write tcp 188.166.225.170:49234->104.36.82.5:4859: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865779663 (85h51m2.588342693s) [noise=true , address=104.36.82.3:4787 , piece_id=HEPAPP2MIARAK5S2NAXJ2WRRQ4JAPBDNC7WHV6TLTE5ZJQ3XJA3A, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:104.36.82.3:4787): protocol: write tcp 188.166.225.170:47318->104.36.82.3:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865849888 (85h51m2.588412918s) [noise=true , address=104.36.82.50:4033 , piece_id=3QTCUPIZQ3BBTKQNCF4WSFJSVOYBAMFOGHQHI5LMHTMDHN73ZNNQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1GhnCxqMdrDsaY3SrRs7VmHbSn8htrbQ3zJy2B5Nb77awMkCR6, address:104.36.82.50:4033): protocol: write tcp 188.166.225.170:56318->104.36.82.50:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865896517 (85h51m2.588459587s) [noise=true , address=104.36.82.10:4093 , piece_id=RKV5XLGZOFZ5C3JT6UVGJARFTVJSFQGVHLBWSEN3VBJD2G2RTYCA, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1SsKS8mvgkHszRNtGrt5bktXsJpbdggkeY4tuiAUni6hZ3FnAw, address:104.36.82.10:4093): protocol: write tcp 188.166.225.170:50298->104.36.82.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865928328 (85h51m2.588491368s) [noise=true , address=104.36.82.41:4778 , piece_id=S5TUTBLCFYC7FSXRFW7WGWX5W6SY7PWYP2UJVQRGRFKIESPGEQWQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: write tcp 188.166.225.170:60016->104.36.82.41:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865957904 (85h51m2.588520945s) [noise=true , address=104.36.82.40:4950 , piece_id=K65YR4JNPKHF5IHQLZELDNDN7XLE3M6BFBPW6BYFT57PKJZIIJIQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: write tcp 188.166.225.170:48976->104.36.82.40:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.865986319 (85h51m2.588549349s) [noise=true , address=104.36.82.53:4147 , piece_id=2NR4CJ3PBMEU4AMX7FPH7W4UYUBGKGNKIBJSCRRFPJBPSK3MZHPQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 188.166.225.170:49980->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866016727 (85h51m2.588579757s) [noise=true , address=104.36.82.11:4117 , piece_id=AATGLC6DBJPUADDVCG5SJJ6N327H2CCKOLJPPNGQDO4SLVJ4JUBQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the, address:104.36.82.11:4117): protocol: write tcp 188.166.225.170:57328->104.36.82.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866029461 (85h51m2.588592501s) [noise=true , address=104.36.82.49:4009 , piece_id=S7VJBFZWSNKGWEYYECSJFFG2Q7EXJZMFIHZ6EIVF3SQWPPLBHFEA, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:12p2hucv6vZHowvgLEgMQLSzkE6ZCdvSQtp7Nzf3r6VpNbDmAAZ, address:104.36.82.49:4009): protocol: write tcp 188.166.225.170:60522->104.36.82.49:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866048177 (85h51m2.588611217s) [noise=true , address=104.36.82.12:4166 , piece_id=ZR6EYWU6XBS4WLA365RZTGSSJ5AKTTYGAFZEC7RB2GFGCMSJAPJA, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: write tcp 188.166.225.170:39434->104.36.82.12:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866068516 (85h51m2.588631556s) [noise=true , address=104.36.82.54:4158 , piece_id=655JUTMWI47DELEN3AK4AGTWBLPGR3DW4SR7KEVG2LJV5KJ23GQQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: write tcp 188.166.225.170:56408->104.36.82.54:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866081861 (85h51m2.588644901s) [noise=true , address=104.36.82.6:4899 , piece_id=J5CMSTDHBGSLDQG2DK6B5E42CONDZBARTS4HB5LWZFXD47CAW2CQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 188.166.225.170:48988->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866096690 (85h51m2.58865973s) [noise=true , address=104.36.82.52:4086 , piece_id=YU7VBFHEPYIVBF2QUEMPATXFJLEFMEJVELBQUNTQGITMKUGNCWBQ, seg_pos={0 0}, upload=82342]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:104.36.82.52:4086): protocol: write tcp 188.166.225.170:54442->104.36.82.52:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:52.866111057 (85h51m2.588674097s) [seg_pos={0 0}, upload=82342]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.234406787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X.....XXX.......X....X.....X..] duration: 6.006305931s FAIL (2025-07-28 01:26:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 6.006329696s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X..................X.........X......X..X.....] duration: 5.798677868s FAIL (2025-07-28 01:25:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.798725047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.245396084s FAIL (2025-07-28 01:25:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ing piece... Jul 28 01:24:53.583739662 (85h49m3.306302692s) [noise=true , address=104.36.82.15:4272 , piece_id=XO4JVYMHXEFBVMIV3YGYJMCS4BISRX43POUZUE56W4E3CSJ4UJBA, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.583780089 (85h49m3.306343129s) [noise=true , address=104.36.82.14:4226 , piece_id=VL33INJ7ZF26CGEPDYYCXEMX4EPPBGWUY3VZP2PHZZXH2YEHJNGQ, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.583992395 (85h49m3.306555425s) [noise=true , address=104.36.82.4:4813 , piece_id=J4GLWUIPE4DOYLZV45DP3SUNDXJ76H5XHPGEISDEXKGIJXBDKY7A, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.584080524 (85h49m3.306643564s) [noise=true , address=104.36.82.12:4173 , piece_id=LVK73ISY4IO544CJ334IMX6LA26PJH56AKJ26ICY4DFBWVU2VWMA, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.584231292 (85h49m3.306794332s) [noise=true , address=104.36.82.7:4927 , piece_id=7I7EKLTFS4LY6U5HSG25P3XO5YGGSJAJKORB7BPN34B5CPMCOQXA, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.584471552 (85h49m3.307034592s) [noise=true , address=104.36.82.55:4197 , piece_id=EY6FM7RKXFSURMLFIJESXBXUAWY5IS7FVJHKJY7PWDYWG5KZP26A, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.584673108 (85h49m3.307236148s) [noise=true , address=104.36.82.56:4229 , piece_id=MY3PK4RTUYW7ENNQD6K7A6APUREH3DZYXYDEXBFXQC7OJ6D77MHA, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.584834977 (85h49m3.307398017s) [noise=true , address=79.127.220.98:4404 , piece_id=VYAUVOHO5PK2C6LH7YLO27P23QHL6CIY4757RGEEH3U4WHPNSCBA, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:53.584900733 (85h49m3.307463763s) [noise=true , address=104.36.82.11:4131 , piece_id=EOAHD7MKFDQ3DNR3JPL6K3VFQCKRYP6DWQQSZM76KDIRYZSOUXBQ, seg_pos={0 0}, upload=82326]: Uploading piece... Jul 28 01:24:54.930934710 (85h49m4.65349775s) [upload=82326]: Next returned nil segment Jul 28 01:24:54.930952555 (85h49m4.653515595s) [upload=82326]: Waiting for error group managing segments... Jul 28 01:24:55.285575573 (85h49m5.008138613s) [noise=true , address=79.127.220.98:4404 , piece_id=VYAUVOHO5PK2C6LH7YLO27P23QHL6CIY4757RGEEH3U4WHPNSCBA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: <nil> Jul 28 01:24:56.442229358 (85h49m6.164792408s) [noise=true , address=79.127.201.215:4354 , piece_id=RUTTT7CY7MT3TZUDXEXWOQ53GCE544HFG3P76QQFFE5QHFPSCE6A, seg_pos={0 0}, upload=82326]: Done uploading piece. err: <nil> Jul 28 01:25:52.392068517 (85h50m2.114631557s) [noise=true , address=104.36.82.42:4822 , piece_id=V77OMYXAOFOT7FJ4HUJ5I46M3MMWO3K24VQFT6VPA3L3HO6ICTGA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:104.36.82.42:4822): protocol: rpc: dial tcp 104.36.82.42:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392132629 (85h50m2.114695669s) [noise=true , address=104.36.82.53:4130 , piece_id=5QNXOID7WJ5Y3C4TUFDHXMNBHBAT5MC7IGFBXR2ZQUCKD6N35K4Q, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: rpc: dial tcp 104.36.82.53:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392191962 (85h50m2.114755002s) [noise=true , address=104.36.82.2:4762 , piece_id=SPBOXEVG2NCIODH3UHZPUYCUVBYNTF34JYBBKHYE4TE7JGSCMHRA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392233192 (85h50m2.114796232s) [noise=true , address=104.36.82.12:4173 , piece_id=LVK73ISY4IO544CJ334IMX6LA26PJH56AKJ26ICY4DFBWVU2VWMA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: rpc: dial tcp 104.36.82.12:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392284991 (85h50m2.114848031s) [noise=true , address=104.36.82.41:4777 , piece_id=TVM62DIOJXCXG7D4COTP2VPPRWZEGR3ECKURJJUWWNDJ4WXXLGQA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt, address:104.36.82.41:4777): protocol: rpc: dial tcp 104.36.82.41:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392317413 (85h50m2.114880453s) [noise=true , address=104.36.82.20:4468 , piece_id=7MTYL7H7A6JW2JDF3QKYELDK4H7CP6RCLG7G2M463KZHBL5VN5PQ, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:4468): protocol: rpc: dial tcp 104.36.82.20:4468: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392365936 (85h50m2.114928976s) [noise=true , address=104.36.82.10:4100 , piece_id=UFKED6X35LWBSKOPK66LGCD66SLUMVR3OYQELBKIGUFDMYX44VZA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: rpc: dial tcp 104.36.82.10:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392587379 (85h50m2.11515042s) [noise=true , address=104.36.82.21:4489 , piece_id=5XJ5HGLVVGEEVUNTJGJJPRDPK4QHSHHUIOWWMUR7AEGGCD7XSPRQ, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: rpc: dial tcp 104.36.82.21:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392707719 (85h50m2.115270749s) [noise=true , address=104.36.82.15:4272 , piece_id=XO4JVYMHXEFBVMIV3YGYJMCS4BISRX43POUZUE56W4E3CSJ4UJBA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:104.36.82.15:4272): protocol: rpc: dial tcp 104.36.82.15:4272: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392756733 (85h50m2.115319773s) [noise=true , address=104.36.82.14:4226 , piece_id=VL33INJ7ZF26CGEPDYYCXEMX4EPPBGWUY3VZP2PHZZXH2YEHJNGQ, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: rpc: dial tcp 104.36.82.14:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392773505 (85h50m2.115336545s) [noise=true , address=104.36.82.7:4927 , piece_id=7I7EKLTFS4LY6U5HSG25P3XO5YGGSJAJKORB7BPN34B5CPMCOQXA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:1AD5Ju87nY6LL5aTbFFsfNJRt8gkyGExsSrJLf6AP2q1nd9HBr, address:104.36.82.7:4927): protocol: rpc: dial tcp 104.36.82.7:4927: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392802550 (85h50m2.11536559s) [noise=true , address=104.36.82.4:4813 , piece_id=J4GLWUIPE4DOYLZV45DP3SUNDXJ76H5XHPGEISDEXKGIJXBDKY7A, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: rpc: dial tcp 104.36.82.4:4813: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392836296 (85h50m2.115399336s) [noise=true , address=104.36.82.55:4197 , piece_id=EY6FM7RKXFSURMLFIJESXBXUAWY5IS7FVJHKJY7PWDYWG5KZP26A, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392895579 (85h50m2.115458619s) [noise=true , address=104.36.82.56:4229 , piece_id=MY3PK4RTUYW7ENNQD6K7A6APUREH3DZYXYDEXBFXQC7OJ6D77MHA, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: write tcp 188.166.225.170:40720->104.36.82.56:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392912340 (85h50m2.11547538s) [noise=true , address=104.36.82.11:4131 , piece_id=EOAHD7MKFDQ3DNR3JPL6K3VFQCKRYP6DWQQSZM76KDIRYZSOUXBQ, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: rpc: dial tcp 104.36.82.11:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392917932 (85h50m2.115480972s) [noise=true , address=104.36.82.52:4106 , piece_id=OHS35KFMFSZEUY7EZ5M2VSS3LNVJPYGWYYRKVFGIDVXLSGDJBRDQ, seg_pos={0 0}, upload=82326]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: write tcp 188.166.225.170:37164->104.36.82.52:4106: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:52.392973689 (85h50m2.115536729s) [seg_pos={0 0}, upload=82326]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:25:52.635841766 (85h50m2.358404806s) [upload=82326]: Done uploading segments... Jul 28 01:25:52.635872876 (85h50m2.358435916s) [upload=82326]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.245416544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................X.....XXX.......X....X.....X.] duration: 5.793203256s FAIL (2025-07-28 01:25:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.793232993s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X..................X.........X......X..X....] duration: 5.955248515s FAIL (2025-07-28 01:25:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.955274664s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)