WEB开发在进行大数据,和高并发难免会碰到服务器瓶颈和数据库瓶颈,这就需要负载了,前端负载,数据库负载均衡。一般情况是前端做反向代理,后端做数据库集群,就是数据库的HA(High Available)。略懂一些,一下是一些个人看法,仅供初级接触的人的一些看法。
一般情况mysql在高数据存储和高并发查询,会做我们常说的“读写分离”,上次某个人跟我念叨了好久的“分库分表”其实就是数据库集群,其实他自己没有把本质说清楚。一般读写分离,就是分主库和从库,也就是master和slave集群,分库有垂直和水平划分两种,按照业务需求自由灵活配置,mysql提供简单易于配置的HA方案。先上一张图,图片从其它地方找的,自己懒得画:
关于mysql 主从配置,了解之后真的超简单下面来个简单的试水:
1.我们先来两个服务器,一个做主,一个做从,一般我就用虚拟机了,这里省略虚拟机的安装过程,我用的是vmvare

楼主用的unbuntu,其它的可自行选择OS。
2,。环境准备阶段,两台都装上mysql。下载mysql客户端,
unbuntu对应的mysql版本在官网是deb格式,deb格式的包安装的时候是会相互依赖的,所以下bundle那个打包在一起的包,然后解压(tar zxvf *.gz),安装的时候用dpkg -i *.deb ,这样就不用去管各个deb的依赖关系了。

mysql安装完成了,检查到了启动进程。

也能正常登陆,两台机器都昨晚此操作之后,开始下一步。主从配置:

我这里有112,108两台服务器,例子上,我们就把112作为master,108作为slave构建主从复制。
进入112的mysql配置:

这里有两个关键配置:打开数据库二进制日志log-bin配置,配置serverID,作为集群的mysql,每台都要有个唯一的不为0的serverid,默认不配置是0.配置同步数据库binlog-do-db等等,character_set_server=utf8这个是用来启动设置字符集的,防止中文乱码的。默认貌似是latin。
从的配置:

大致差不多。注意主从的数据库映射,这里两边都是建了一个叫project的SCHEMA
在做HA之前,先确保环境正确,让两台mysql相互访问没有问题。
1.关闭两台机器的放火墙 unbuntu的是ufw disable。
2.给予本地mysql服务器允许远程用户访问。一劳永逸就这样,楼主真的很懒,没有做什么用户角色划分。全是用root。

用root进入控制台,进入mysql schema,讲user表中root对应这条记录的host改为%,这样root用户就能以任何IP登录到这台服务器了。两台服务器都做这个操作。

做完之后确保端口什么的都能访问。
接下来在从数据库执行:
change master to master_host='192.168.11.112',master_user='root',master_password='111111',master_log_file='mysql-bin.000005',master_log_pos=154;
建立映射关系。
这里master_log_pos如果有问题。可以去master查看,然后再做同步:

这里可以看到主服务器的日志文件名字,postion。如果不行,重新做上面映射,在做之前可以先停止从服务器的同步。

重新按照master的位置执行change master 。。。之后再启动slave 同步

观察日志是否正常:

测试主从同步:
112主数据库创建一张表,去从数据库查看
从数据库可以看到了:

增删改就不一一列举了。
下面看一个简单的springmvc的项目构建:maven快速构建一个springmvc项目

新建一个maven工程,配置POM。楼主,用的东西比较多,杂,当做模板工程,是个大杂烩,spring,hibernate,mybatis,kafka,activemq,c3p0,drui各种乱七八糟的都拿来了。
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>lopsaweb</groupId>
<artifactId>lopsa</artifactId>
<packaging>war</packaging>
<version>1.0.0</version>
<name>lopsa Maven Webapp</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<springversion>3.1.1.RELEASE</springversion>
<junitversion>3.8.1</junitversion>
<!-- mybatis版本号 -->
<mybatis.version>3.2.4</mybatis.version>
</properties>
<dependencies>
<!-- <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId>
<version>${junitversion}</version> <scope>test</scope> </dependency> -->
<!-- https://mvnrepository.com/artifact/javax.servlet/servlet-api <dependency>
<groupId>javax.servlet</groupId> <artifactId>servlet-api</artifactId> <version>2.5</version>
</dependency> -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.javassist/javassist -->
<dependency>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
<version>3.20.0-GA</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.hibernate/hibernate-core -->
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>3.6.0.Final</version>
</dependency>
<!-- spring hibernate 整合包 -->
<!-- https://mvnrepository.com/artifact/org.springframework/spring-orm -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<!-- mybatis核心包 -->
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>${mybatis.version}</version>
</dependency>
<!-- mybatis/spring包 -->
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
<version>1.2.2</version>
</dependency>
<!-- mysql驱动包 -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.29</version>
</dependency>
<!--druid -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.0.11</version>
</dependency>
<!-- https://mvnrepository.com/artifact/c3p0/c3p0 -->
<dependency>
<groupId>c3p0</groupId>
<artifactId>c3p0</artifactId>
<version>0.9.1.2</version>
</dependency>
<!-- apache activemq -->
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>5.14.1</version>
</dependency>
<!--apache kafka -->
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10 -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.10.0.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.9</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.8.3</version>
</dependency>
</dependencies>
<build>
<finalName>lopsa</finalName>
</build>
</project>
xml: 配置1 springmvc页面映射,数据库事务,注解扫描等等
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop.xsd " >
<!-- 开启注解 -->
<bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter" />
<bean id="viewResolver"
class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<property name="prefix" value="/WEB-INF/view/" />
<property name="suffix" value=".jsp" />
</bean>
<context:component-scan base-package="com.mvcpro" />
<!-- 对数据源进行事务管理 -->
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager"
p:dataSource-ref="druidDataSource_local" />
<!-- 用注解来实现事务管理 -->
<tx:annotation-driven transaction-manager="transactionManager" proxy-target-class="true"/>
</beans>
xml配置2:hibernate相关配置
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop.xsd ">
<!-- hibernate 相关 -->
<!-- Hibernate session factory -->
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource">
<ref bean="druidDataSource_master" />
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.hbm2ddl.auto">create</prop>
</props>
</property>
<property name="annotatedClasses">
<list>
<value>com.mvcpro.hibernate.entity.User</value>
</list>
</property>
</bean>
</beans>
xml配置3:mybatis相关
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop.xsd " >
<!-- 注入从数据库 用来读取数据 mybatis文件配置,扫描所有mapper文件-->
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean"
p:dataSource-ref="druidDataSource_slave" p:configLocation="/WEB-INF/mybatis/mybatis-config.xml"
p:mapperLocations="classpath:com/mvcpro/model/*.xml" />
<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
<constructor-arg index="0" ref="sqlSessionFactory" />
</bean>
</beans>
基本构思,是hibernate用来写主库,mybatis用来读从库;其实为了分散压力,可能会用hash用到多个master库对应多个slave库,写库通过hash算法分散到master数据库集群,读库通过hash检索到slave库集群,这大概就是数据库读写分离了吧。
小应用:
为了实现两台机器同步执行脚本,利用activemq队列消息机制做了一个同步脚本,首先写个接受消息的客户端,队列可配置:
GetMessage.java
package salveorder;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
import javax.jms.Connection;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.ObjectMessage;
import javax.jms.Session;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.springframework.beans.BeanUtils;
import entity.LinuxMessage;
/**
* 接受命令消息
* @author Henry
*
*/
public class GetMessage extends Thread {
private static String quename;
@Override
public void run(){
System.out.println("正在接收队列:"+quename);
while(true){
receiveMessage(this.quename);
}
}
public void receiveMessage(String quename){
//ArrayList<LinuxMessage> list=new ArrayList<>();
//LinuxMessage messageet=new LinuxMessage();
try{
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://192.168.11.106:61616");
connectionFactory.setTrustedPackages(new ArrayList(Arrays.asList("entity".split(","))));
Connection connection = connectionFactory.createConnection();
connection.start();
final Session session = connection.createSession(Boolean.TRUE, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue(quename);
MessageConsumer consumer = session.createConsumer(destination);
//listener 方式
consumer.setMessageListener(new MessageListener() {
public void onMessage(Message msg) {
ObjectMessage message = (ObjectMessage) msg;
//TODO something....
try {
// User user = (User) message.getObject();
LinuxMessage lxmessage= (LinuxMessage) message.getObject();
System.out.println("收到消息:"+lxmessage.getMessage());
//
ExcOrderService.excuteLinuxOrderRuntime(lxmessage);
System.out.println("处理:"+lxmessage.getMessage());
} catch (Exception e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
session.commit();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});
TimeUnit.MINUTES.sleep(1);
session.close();
connection.close();
}catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void main(String[] args){
Properties p = new Properties();
try {
InputStream in = new BufferedInputStream(new FileInputStream("D:/dev/allspace/OrderExc/src/salveorder/config.properties"));
//InputStream in = ClassLoader.getSystemResourceAsStream("D:/dev/allspace/OrderExc/src/salveorder/config.properties");
//InputStream in = new BufferedInputStream(new FileInputStream("/home/app/OrderExc_lib/config.properties"));
p.load(in);
quename=(String) p.get("queueName");
GetMessage gtmessage=new GetMessage();
//gtmessage.receiveMessage();
new Thread(gtmessage).run();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
其实这里IP也可以做成配置的:
package salveorder;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import entity.LinuxMessage;
public class ExcOrderService {
public static void excuteLinuxOrder(LinuxMessage message) {
ProcessBuilder processBuilder = new ProcessBuilder(message.getMessage());
// ...其它参数添加
// processBuilder.command(commands);
// processBuilder.directory(path);//切换工作目录
processBuilder.redirectErrorStream(true);
try {
Process process = processBuilder.start();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void excuteLinuxOrderRuntime(LinuxMessage message) {
Process ps;
try {
ps = Runtime.getRuntime().exec(message.getMessage());
System.out.print(loadStream(ps.getInputStream()));
System.err.print(loadStream(ps.getErrorStream()));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
static String loadStream(InputStream in) throws IOException {
int ptr = 0;
in = new BufferedInputStream(in);
StringBuffer buffer = new StringBuffer();
while( (ptr = in.read()) != -1 ) {
buffer.append((char)ptr);
}
return buffer.toString();
}
}
ExcOrderService.java 调用linux本地shell脚本
package salveorder;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import entity.LinuxMessage;
public class ExcOrderService {
public static void excuteLinuxOrder(LinuxMessage message) {
ProcessBuilder processBuilder = new ProcessBuilder(message.getMessage());
// ...其它参数添加
// processBuilder.command(commands);
// processBuilder.directory(path);//切换工作目录
processBuilder.redirectErrorStream(true);
try {
Process process = processBuilder.start();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void excuteLinuxOrderRuntime(LinuxMessage message) {
Process ps;
try {
ps = Runtime.getRuntime().exec(message.getMessage());
System.out.print(loadStream(ps.getInputStream()));
System.err.print(loadStream(ps.getErrorStream()));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
static String loadStream(InputStream in) throws IOException {
int ptr = 0;
in = new BufferedInputStream(in);
StringBuffer buffer = new StringBuffer();
while( (ptr = in.read()) != -1 ) {
buffer.append((char)ptr);
}
return buffer.toString();
}
}
LinuxMessage.java 传输对象
package entity;
import java.io.Serializable;
public class LinuxMessage implements Serializable{
/**
*
*/
private static final long serialVersionUID = 1L;
private String message;
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
}
config.properties 队列名称配置
比如112,108配置两个不同的队列,分别给这两个队列发送脚本命令,比如启动两台机器的tomcat。
#activemq topic
queueName=Linux1
代码打包成执行jar
讲脚本命令编辑成可执行文件 放到目录中,注意文件的权限


activemq的安装启动,楼主是装在本地windows上的,下载zip包解压,更改配置:
更改activemq.xml里面的tcp监听不然只能监听到本地的请求:

启动activemq服务

运行112,108打包的接收消息服务jar

从windows通过activmq发送消息
执行启动112,108的tomcat,同理实现远程执行脚本。
package masterOrder;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.DeliveryMode;
import javax.jms.Destination;
import javax.jms.MessageProducer;
import javax.jms.ObjectMessage;
import javax.jms.Session;
import org.apache.activemq.ActiveMQConnectionFactory;
import entity.LinuxMessage;
public class SendMessage {
public void sendMessage(LinuxMessage message){
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://127.0.0.1:61616");
Connection connection;
try {
connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(Boolean.TRUE, Session.AUTO_ACKNOWLEDGE);
Destination destination1 = session.createQueue("Linux1.foo");
Destination destination2 = session.createQueue("Linux2.foo");
MessageProducer producer1 = session.createProducer(destination1);
MessageProducer producer2 = session.createProducer(destination2);
producer1.setDeliveryMode(DeliveryMode.PERSISTENT);
producer2.setDeliveryMode(DeliveryMode.PERSISTENT);
ObjectMessage messageobj = session.createObjectMessage();
messageobj.setObject(message);
//
producer1.send(messageobj);
producer2.send(messageobj);
session.commit();
session.close();
connection.close();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void main(String[] args){
SendMessage sdmessage=new SendMessage();
LinuxMessage message=new LinuxMessage();
//message.setMessage("sh startup.sh;");
//message.setMessage("./stopTomcat.sh");
message.setMessage("./startTomcat.sh");
//message.setMessage("cd /home/sofvare/apache-tomcat-7.0.59/bin/;pwd;");
sdmessage.sendMessage(message);
}
}
ok,成功执行了!
另外上传项目的时候,可以把112和108做个共享存储,做mount远程文件夹,两台机器都要装nfs服务,unbuntu装起来非常简单 apt-get install nfs-common、service nfs-kernel-server restart,这样就只用上传一次即可,tomcat链路都指向共享存储路径,楼主讲108的/home/app/project 挂到了112的/home/app/project 下了,这样上传文件只用上传一份即可,理想情况可将112在和本地eclipse的编译环境做个共享推送,这样实现本地编译,三机一键启动。

nginx反向代理:
下载nginx 包:tar 解压

./configure; make ; make install;三步走
安装完成,配置:

杀掉nginx 重启

观察日志访问
以上都是简单配置,更多详细配置,根据需求配置。
本文介绍MySQL主从配置步骤及注意事项,并演示如何使用Maven快速构建SpringMVC项目,涵盖POM配置、XML配置以及数据库读写分离策略。
372

被折叠的 条评论
为什么被折叠?



