Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 1.908188006s FAIL (2025-07-28 02:05:59 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.908225989s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X......X..........................................X.....] duration: 5.757762425s FAIL (2025-07-28 02:05:52 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.75779137s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration: 5.991299053s FAIL (2025-07-28 02:05:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 5.991337427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 1.913185056s FAIL (2025-07-28 02:05:09 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.913216146s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration: 7.896956286s FAIL (2025-07-28 02:04:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.896983016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X......X..........................................X....] duration: 5.788523913s FAIL (2025-07-28 02:04:38 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.788548981s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration: 9.832154277s FAIL (2025-07-28 02:04:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.83218617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 1.909455421s FAIL (2025-07-28 02:04:07 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.909488214s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration: 7.26779368s FAIL (2025-07-28 02:04:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.267823078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.029071141s FAIL (2025-07-28 02:03:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.029100106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.....X......X..........................................X...] duration: 5.756934983s FAIL (2025-07-28 02:03:27 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.756965211s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration: 8.249183048s FAIL (2025-07-28 02:03:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 8.249261969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.88033261s FAIL (2025-07-28 02:02:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.880366004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 1.847182705s FAIL (2025-07-28 02:02:53 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.847212201s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.....X......X..........................................X..] duration: 5.848186695s FAIL (2025-07-28 02:02:16 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.848221592s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration: 8.403599482s FAIL (2025-07-28 02:02:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 8.40362921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 1.834208903s FAIL (2025-07-28 02:02:08 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.834234203s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.....X......X..........................................X.] duration: 5.795448211s FAIL (2025-07-28 02:01:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.795472578s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.87323916s FAIL (2025-07-28 02:01:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.873263838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration: 7.339993581s FAIL (2025-07-28 02:01:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.340028389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.....X......X..........................................X] duration: 4.080133937s FAIL (2025-07-28 02:01:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 4.080155058s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration: 7.378314419s FAIL (2025-07-28 02:01:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.378336892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 5.781203987s FAIL (2025-07-28 02:01:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.781227682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 1.829774027s FAIL (2025-07-28 02:01:00 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.829804745s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....X......X..........................................] duration: 5.778944084s FAIL (2025-07-28 02:00:57 UTC) orbiter warning: uplink: too many requests Duration: 5.778972237s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration: 8.39567377s FAIL (2025-07-28 02:00:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 8.395702074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X.....X......X.........................................] duration: 6.244614058s FAIL (2025-07-28 02:00:27 UTC) orbiter warning: uplink: too many requests Duration: 6.244638665s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 5.762841175s FAIL (2025-07-28 02:00:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.762869187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration: 7.553172766s FAIL (2025-07-28 02:00:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.553221349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..X......................................................] duration: 5.778765821s FAIL (2025-07-28 02:00:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.778814493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 1.814164641s FAIL (2025-07-28 01:59:47 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.814189199s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X.....X......X........................................] duration: 5.760431542s FAIL (2025-07-28 01:59:37 UTC) orbiter warning: uplink: too many requests Duration: 5.760466418s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX........................................] duration: 9.427728572s FAIL (2025-07-28 01:59:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.42775349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration: 5.892283495s FAIL (2025-07-28 01:59:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.892311769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X....X.....X......X.......................................] duration: 5.854472719s FAIL (2025-07-28 01:59:12 UTC) orbiter warning: uplink: too many requests Duration: 5.854494983s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration: 8.646214428s FAIL (2025-07-28 01:58:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 8.646250857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 1.81402284s FAIL (2025-07-28 01:58:56 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.814049831s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..X....................................................] duration: 6.125123659s FAIL (2025-07-28 01:58:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.125147374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X....X.....X......X......................................] duration: 5.670588186s FAIL (2025-07-28 01:58:21 UTC) orbiter warning: uplink: too many requests Duration: 5.670611721s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration: 9.486182074s FAIL (2025-07-28 01:58:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.486209868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 1.810796916s FAIL (2025-07-28 01:57:56 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.810827957s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration: 5.865188794s FAIL (2025-07-28 01:57:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.865207799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X....X.....X......X.....................................] duration: 5.641228712s FAIL (2025-07-28 01:57:35 UTC) orbiter warning: uplink: too many requests Duration: 5.641260724s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration: 6.952899336s FAIL (2025-07-28 01:57:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 6.95292298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..X..................................................] duration: 5.816386572s FAIL (2025-07-28 01:56:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.816418724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 1.878453546s FAIL (2025-07-28 01:56:48 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.878492399s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 7.422343783s FAIL (2025-07-28 01:56:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.422359593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X....X.....X......X....................................] duration: 5.994079902s FAIL (2025-07-28 01:56:28 UTC) orbiter warning: uplink: too many requests Duration: 5.994098888s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X......X..X.................................................] duration: 5.797878852s FAIL (2025-07-28 01:56:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.7979049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X......X..X................................................] duration: 5.887521444s FAIL (2025-07-28 01:56:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.887544228s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)