Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 8.189765474s FAIL (2025-07-28 01:56:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 8.189793568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 1.846067852s FAIL (2025-07-28 01:55:57 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.846092168s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X....X.....X......X...................................] duration: 5.707271435s FAIL (2025-07-28 01:55:30 UTC) orbiter warning: uplink: too many requests Duration: 5.707297534s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X......X..X...............................................] duration: 5.85837573s FAIL (2025-07-28 01:55:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.858403373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X....X.....X......X..................................] duration: 5.795444473s FAIL (2025-07-28 01:55:12 UTC) orbiter warning: uplink: too many requests Duration: 5.795471715s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 1.840462568s FAIL (2025-07-28 01:55:06 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.840487785s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.......X....X.....X......X.................................] duration: 5.778982848s FAIL (2025-07-28 01:54:55 UTC) orbiter warning: uplink: too many requests Duration: 5.77900508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 7.830240505s FAIL (2025-07-28 01:54:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.830260014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X......X..X..............................................] duration: 5.837185003s FAIL (2025-07-28 01:54:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.837212266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.................................] duration: 7.75144525s FAIL (2025-07-28 01:54:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.751463665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.......X....X.....X......X................................] duration: 5.907580092s FAIL (2025-07-28 01:54:16 UTC) orbiter warning: uplink: too many requests Duration: 5.907603306s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 1.852851671s FAIL (2025-07-28 01:53:53 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.85287721s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX.......X....X.....X......X...............................] duration: 5.89884027s FAIL (2025-07-28 01:53:36 UTC) orbiter warning: uplink: too many requests Duration: 5.898876971s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 1.906525655s FAIL (2025-07-28 01:53:30 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.906555552s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X......X..X.............................................] duration: 5.752348899s FAIL (2025-07-28 01:53:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.75237639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX................................] duration: 7.305735806s FAIL (2025-07-28 01:53:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.305768147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXX.......X....X.....X......X..............................] duration: 5.866628981s FAIL (2025-07-28 01:53:05 UTC) orbiter warning: uplink: too many requests Duration: 5.866659829s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X......X..X............................................] duration: 5.830796141s FAIL (2025-07-28 01:53:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.830825758s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X......X..X...........................................] duration: 5.872925489s FAIL (2025-07-28 01:52:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872950567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 1.851211935s FAIL (2025-07-28 01:52:39 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.851241481s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..XXX.......X....X.....X......X.............................] duration: 5.853207908s FAIL (2025-07-28 01:52:24 UTC) orbiter warning: uplink: too many requests Duration: 5.853237554s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............................] duration: 9.245546615s FAIL (2025-07-28 01:52:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.245584448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...XXX.......X....X.....X......X............................] duration: 5.80348604s FAIL (2025-07-28 01:51:57 UTC) orbiter warning: uplink: too many requests Duration: 5.803511617s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 1.839896334s FAIL (2025-07-28 01:51:51 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.839921642s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X......X..X..........................................] duration: 5.864142563s FAIL (2025-07-28 01:51:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.864169634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............................] duration: 7.680169231s FAIL (2025-07-28 01:51:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 7.680196963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....XXX.......X....X.....X......X...........................] duration: 5.789935123s FAIL (2025-07-28 01:50:57 UTC) orbiter warning: uplink: too many requests Duration: 5.789960693s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X......X..X.........................................] duration: 6.044274877s FAIL (2025-07-28 01:50:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.044309594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 1.864089665s FAIL (2025-07-28 01:50:39 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.864120773s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............................] duration: 9.123902484s FAIL (2025-07-28 01:50:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.123928365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X......X..X........................................] duration: 6.358028731s FAIL (2025-07-28 01:49:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.358059721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....XXX.......X....X.....X......X..........................] duration: 5.924211554s FAIL (2025-07-28 01:49:51 UTC) orbiter warning: uplink: too many requests Duration: 5.924240029s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 1.872021198s FAIL (2025-07-28 01:49:36 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.872059992s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........X......X..X.......................................] duration: 5.770577907s FAIL (2025-07-28 01:49:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770609086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............................] duration: 9.849016269s FAIL (2025-07-28 01:49:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.849034474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 1.857650114s FAIL (2025-07-28 01:49:02 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.857703295s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.....XXX.......X....X.....X......X.........................] duration: 5.778482329s FAIL (2025-07-28 01:49:00 UTC) orbiter warning: uplink: too many requests Duration: 5.778508378s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 9.697361443s FAIL (2025-07-28 01:48:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 9.697416027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.....XXX.......X....X.....X......X........................] duration: 5.748616265s FAIL (2025-07-28 01:48:23 UTC) orbiter warning: uplink: too many requests Duration: 5.748641923s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........X......X..X......................................] duration: 5.824304268s FAIL (2025-07-28 01:48:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.824327855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 1.814147201s FAIL (2025-07-28 01:48:08 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.814174514s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 1.913653245s FAIL (2025-07-28 01:47:39 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.913684203s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration: 10.705754395s FAIL (2025-07-28 01:47:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 10.705780323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........X......X..X.....................................] duration: 5.804861606s FAIL (2025-07-28 01:47:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.804886955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.....XXX.......X....X.....X......X.......................] duration: 5.866815213s FAIL (2025-07-28 01:47:25 UTC) orbiter warning: uplink: too many requests Duration: 5.866840082s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration: 10.761520414s FAIL (2025-07-28 01:47:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 10.761543459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 1.84081333s FAIL (2025-07-28 01:46:38 UTC) orbiter warning: digest mismatch: after a70e411f3caec8bac58bdadf610b78ca4fd35f5b6d1a7c1d92126c4dd558248a != before 27b97e36cacba9a22bf94ba2f7b134b10243007531dc7c88d5252eea05c7c65d Duration: 1.840841134s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........X......X..X....................................] duration: 5.834465929s FAIL (2025-07-28 01:46:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=NEQ4NZNMCAUAW5YERDV45KZ46OWTTBH3AW2PJYZZO6Y7B2NZSLWQ, seg_pos={0 0}, upload=82277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:39540->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:22:02.496636431 (85h46m12.219199471s) [seg_pos={0 0}, upload=82277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:22:02.496706014 (85h46m12.219269054s) [upload=82277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:22:02.496710042 (85h46m12.219273082s) [upload=82277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:22:02.788878957 (85h46m12.511441987s) [upload=82277]: Done uploading segments... Jul 28 01:22:02.788897482 (85h46m12.511460522s) [upload=82277]: Upload finished. err: <nil> Jul 28 01:22:33.340997726 (85h46m43.063560776s) [upload=82286]: Starting upload Jul 28 01:22:33.341078010 (85h46m43.06364105s) [upload=82286]: Done starting upload Jul 28 01:22:33.341161048 (85h46m43.063724089s) [upload=82286]: Uploading segments... Jul 28 01:22:33.341327988 (85h46m43.063891048s) [upload=82286]: Got next segment. Inline: false Jul 28 01:22:33.341388954 (85h46m43.063951994s) [upload=82286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:22:33.341396259 (85h46m43.063959299s) [upload=82286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:22:33.571381107 (85h46m43.293944147s) [seg_pos={0 0}, upload=82286]: Begin upload segment... Jul 28 01:22:33.571421374 (85h46m43.293984414s) [seg_pos={0 0}, upload=82286]: Done begin upload segment. Jul 28 01:22:33.571440630 (85h46m43.29400367s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571510815 (85h46m43.294073855s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.571815388 (85h46m43.294378428s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572519035 (85h46m43.295082075s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572636028 (85h46m43.295199058s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.572898590 (85h46m43.29546163s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573214315 (85h46m43.295777355s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.573744779 (85h46m43.296307819s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574022701 (85h46m43.296585731s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.574480828 (85h46m43.297043868s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575057832 (85h46m43.297620872s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575414985 (85h46m43.297978025s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.575884543 (85h46m43.298447583s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576100497 (85h46m43.298663537s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.576618267 (85h46m43.299181308s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:33.577451201 (85h46m43.300014241s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Uploading piece... Jul 28 01:22:35.337557229 (85h46m45.060120269s) [upload=82286]: Next returned nil segment Jul 28 01:22:35.337574663 (85h46m45.060137703s) [upload=82286]: Waiting for error group managing segments... Jul 28 01:22:35.615284824 (85h46m45.337847864s) [noise=true , address=77.90.31.15:4698 , piece_id=BW5N5HJAZ3BADCCU5YRGUH6AEWNJWL4IGRQBMKBSHQBJXAKYUVAA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.662676818 (85h46m45.385239848s) [noise=true , address=45.91.116.4:4940 , piece_id=VIAL6F64UBW5ZB47N44XXO6TSC4WCQR4IQZ2MR52OFF2GFX4T6GA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.663128693 (85h46m45.385691723s) [noise=true , address=77.90.31.13:4627 , piece_id=URIQMVY34JW42CJGXWNZYTMVS7QWQVYRKHZTGQLHTLIDZPOXOEPQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.678658375 (85h46m45.401221415s) [noise=true , address=45.91.116.6:4975 , piece_id=JW74D3TCRXTSDE6HIJWNQ6JT3LM64DGERDU6ZLLNCJEVL77EUX3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.702262036 (85h46m45.424825076s) [noise=true , address=45.91.116.14:4190 , piece_id=XGIAUUZAVGYP6YTOUWQELY5LDBEPULHF6ZBGQDTQT5QIFVJVUS3Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.703011440 (85h46m45.425574481s) [noise=true , address=77.90.31.19:4843 , piece_id=VSHIEUWYLEQ5AGDQ2GIP5RJFRSO5CTDUXMEJT2VMRDNBPDBDTZPA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.721580278 (85h46m45.444143318s) [noise=true , address=45.91.116.12:4226 , piece_id=D43OFUMHIL56JG2RSBJ3CEWQT5ANE7CMLOTAKWOSNHOQ4S5IF5RA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.771151853 (85h46m45.493714913s) [noise=true , address=45.91.116.6:4974 , piece_id=CLX5UP636RC7IT6JEESEFAYKLMCCJNO43I52GEX4VNKU2FUI5YOA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.775572440 (85h46m45.498135481s) [noise=true , address=45.91.116.11:4119 , piece_id=WP4I3E3SJEZXYEOP7JICNXS4NEQHHGLXE53HWG5FVOYHRNS2GJKQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.783070266 (85h46m45.505633306s) [noise=true , address=45.91.116.13:4155 , piece_id=7OYS7ESPMXZPIZCO23VC3O774FU4QDKTFV7KX4XI7D52XEQCTK6Q, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.785519603 (85h46m45.508082663s) [noise=true , address=77.90.31.14:4661 , piece_id=LKOTBEWROHTYMBOFZGHVDKXFQEHORIY7JLEZED3S5NYEQIIQDVZA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:22:35.793164820 (85h46m45.51572788s) [noise=true , address=77.90.31.21:4878 , piece_id=MHTXISNOWNTCG2YI3WGDWDI5EA3GHSSL3SE4OQUTNOWXX25MCCEQ, seg_pos={0 0}, upload=82286]: Done uploading piece. err: <nil> Jul 28 01:23:02.703218831 (85h47m12.425781911s) [noise=true , address=104.36.82.9:4005 , piece_id=AGPJBQXUV7NOVDYXQTIDX7RJVN4RLYZDBKYWHKWCJLC3327X5FVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703317860 (85h47m12.4258809s) [noise=true , address=104.36.82.9:4006 , piece_id=MGJ66WSXW2DUFXZF7N3ML5IRCBKLTQWBDYRCLXLY42C7UKWUHEVA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703405257 (85h47m12.425968297s) [noise=true , address=104.36.82.13:4187 , piece_id=LMSKNGGU7ILT3CFMYKQPERHX4Q3K5PKV4KGNMLDLY6V7AVQMJF3A, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703460553 (85h47m12.426023614s) [noise=true , address=104.36.82.3:4767 , piece_id=HP6BY4QSN2RMEOTIJ4TGVD2M3S5EXJIY5WNA227Y6S7JWIHUD5YA, seg_pos={0 0}, upload=82286]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:23:02.703493216 (85h47m12.426056286s) [seg_pos={0 0}, upload=82286]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:23:02.932166987 (85h47m12.654730037s) [upload=82286]: Done uploading segments... Jul 28 01:23:02.932197765 (85h47m12.654760806s) [upload=82286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.834497981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.....XXX.......X....X.....X......X......................] duration: 5.826826073s FAIL (2025-07-28 01:46:27 UTC) orbiter warning: uplink: too many requests Duration: 5.826859097s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration: 10.788929735s FAIL (2025-07-28 01:46:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ... Jul 28 01:27:54.937646588 (85h52m4.660209628s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937854195 (85h52m4.660417235s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.937977431 (85h52m4.660540471s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938271533 (85h52m4.660834573s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938544044 (85h52m4.661107074s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:54.938831725 (85h52m4.661394755s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Uploading piece... Jul 28 01:27:56.426451578 (85h52m6.149014618s) [upload=82375]: Next returned nil segment Jul 28 01:27:56.426469592 (85h52m6.149032632s) [upload=82375]: Waiting for error group managing segments... Jul 28 01:27:57.075790653 (85h52m6.798353693s) [noise=true , address=79.127.201.215:4355 , piece_id=ND7CLNKCVX62C4KE75SB2VIVFQHFF23NBQVXOK7FWDHYSDUIIZ5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: <nil> Jul 28 01:28:53.836561840 (85h53m3.55912488s) [noise=true , address=104.36.82.15:4276 , piece_id=B5525I3JT6OSHQJFLIH225EWZMKYWQCHFVNVHYP3QRFOKK6XH4GQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836622366 (85h53m3.559185406s) [noise=true , address=104.36.82.42:4798 , piece_id=YGMNXUVFQNQ442CSTBVH3VJBBH3V6AD747BUF23UZX764RX2TU5Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:48244->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836638727 (85h53m3.559201777s) [noise=true , address=104.36.82.11:4127 , piece_id=DIJ4KS7IZXYM2LBFWJCIRI547U2KJEYYPSP5NJQXUNCHOJPGPYVA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836672622 (85h53m3.559235672s) [noise=true , address=104.36.82.47:4548 , piece_id=TQV64V5FUEWTS7XS4V3AMDXRJIALRODONLD3V4U6DJ6JSI63UL2A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836689895 (85h53m3.559252945s) [noise=true , address=104.36.82.55:4218 , piece_id=PFGIAFQNL6POXSCZ7NP43WALSLCQVFQUJ2YGLFHJQXFUTSVDMMYA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:55108->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836749388 (85h53m3.559312438s) [noise=true , address=104.36.82.20:4463 , piece_id=4IQMS2DVE2QU7NFEAF6DHVAZ23RP7Z6S7U2QD3GHDUKAXIVUUZTA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 188.166.225.170:34854->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836793463 (85h53m3.559356513s) [noise=true , address=104.36.82.14:4228 , piece_id=POHC2K2WRTSG3NRQ6S2MVCBHXGJ6WEHLHGPQDYH7VGIAM6VEP3NA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 188.166.225.170:60500->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836819653 (85h53m3.559382703s) [noise=true , address=104.36.82.5:4853 , piece_id=T72C4NBVYBADKDCIIDFUCXQ7XSU54W4LKZTRLH7JRWO5DGAE7Y5A, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:44610->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836841626 (85h53m3.559404666s) [noise=true , address=104.36.82.46:4494 , piece_id=2JUHDBQ4W2TTT2RQHXL2CYLX3JBANAKC3CLMEQWH45ZYGBCJ6AGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: write tcp 188.166.225.170:35166->104.36.82.46:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836864630 (85h53m3.55942768s) [noise=true , address=104.36.82.44:4891 , piece_id=CTKYD7K7XDKC3CVTZHVN7MF6EIKKPDAOASNF4PNW2TIE63JGJIXA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: write tcp 188.166.225.170:35168->104.36.82.44:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836891992 (85h53m3.559455032s) [noise=true , address=104.36.82.10:4098 , piece_id=MQRGBLHQIJ3WFYZJL6UYIMGGFYANRP7ORYDXLZXUU2ZRFLGSL3DQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:36722->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836903995 (85h53m3.559467045s) [noise=true , address=104.36.82.41:4764 , piece_id=WXUEZHEVGDKOVP7FI4G7ZUGGNFHGVH2AGM7QDZGBTNBBOB37BC6Q, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836936897 (85h53m3.559499947s) [noise=true , address=104.36.82.2:4734 , piece_id=VFUHEU7XAWYBIWGHK6CPROZPBYNBTXXBBLNCELL7WOG56BF53UGA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:104.36.82.2:4734): protocol: write tcp 188.166.225.170:48348->104.36.82.2:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836944662 (85h53m3.559507702s) [noise=true , address=104.36.82.54:4176 , piece_id=EXKQ6D5VC6M7DZGIEWNSKMFK72OKPXH566O4XQZY5CJ3QV2Z72LQ, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 188.166.225.170:55032->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836982405 (85h53m3.559545455s) [noise=true , address=104.36.82.12:4167 , piece_id=AYPQGD3F5J37ZHCMXWR62W7MOJKHGGOQ3JQ4IV5VH36LW47YRMPA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:54314->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836988947 (85h53m3.559551997s) [noise=true , address=104.36.82.53:4156 , piece_id=X54VQO24I3ZNDBPPLLEWKB23TE3C4LMMLWOWKEECHDR7GEO6C4TA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:49604->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.836994558 (85h53m3.559557608s) [noise=true , address=104.36.82.4:4833 , piece_id=LD2QI3VHEBD6NGL7XVCAKRVQJM6BJA5A6DMFIBMOMIWRIJOJSIFA, seg_pos={0 0}, upload=82375]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: rpc: dial tcp 104.36.82.4:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:53.837022251 (85h53m3.559585301s) [seg_pos={0 0}, upload=82375]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:28:54.072085566 (85h53m3.794648606s) [upload=82375]: Done uploading segments... Jul 28 01:28:54.072110755 (85h53m3.794673785s) [upload=82375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 10.788949373s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)