基于大数据的日志服务的搭建

最新推荐文章于 2022-07-08 15:02:43 发布

原创最新推荐文章于 2022-07-08 15:02:43 发布 · 置顶 · 2.1k 阅读

7 ·

CC 4.0 BY-SA版权

文章标签：

#日志 #hadoop

Hadoop 同时被 3 个专栏收录

4 篇文章

订阅专栏

Flume

2 篇文章

订阅专栏

Spark

1 篇文章

订阅专栏

本文详细介绍了一种基于Hadoop、Flume、log4j的日志采集系统，涵盖日志格式化、清洗、存储至HDFS及HBase，以及通过Spark导入SQL Server的过程。系统还提供了日志查询接口，实现了日志的高效管理和分析。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一. 背景及需要实现的功能：

1. 按照固定格式采集日志【Hadoop+Flume+log4j进行数据采集】

2. 日志数据清洗【定时器+MapReduce+Java/Spark+Scala，从Hadoop读取数据并处理后，存储到Hbase】

3. 数据查询接口【以服务的方式提供数据查询接口】

4. 日志服务部署【需要部署到linux上】

二. 架构：

三. hadoop和Flume的搭建，可以参考我的另外一篇文章：

https://blog.youkuaiyun.com/sunroyi666/article/details/83374983

这里简单记录一下CentOS上Hadoop相关工具搭建过程中的一些命令和经验：

1. Flume相关命令：

设为后台进程（不受ssh远程客户端关闭影响）：

setsid flume-ng agent --name flume-log-agent --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/flume-log-agent.conf -Dflume.root.logger=INFO,console

查看flume进程：

ps -ef |grep flume

删除进程：

kill 进程id

删除端口进程：

lsof -i :44444|grep -v "PID"|awk '{print "kill -9",$2}'|sh

查看端口：

netstat -lnp|grep 88

2. Hadoop命令：

查看目录下文件数量：

hadoop fs -count /smphbeatslog

显示目录下所有文件：

hadoop fs -ls /smphbeatslog

3.统一的日志格式：[format:1][app:smph_beats][userId:1][userName:thinkgem][interFace:/exam/user/save]…

format	日志格式的版本
app	调用类型如：[app:ExamCenter]
type	日志类型如：[type:sql][type:interfaceIn][type:interfaceOut][type:exception]
userAgent	客户端
userId	用户ID
userName	用户名
insertDate	调用日期
domain	域名
interFace	接口
interFaceParam	接口参数
success	1：成功 0：失败
message	调用信息
parameter	用户自定义参数：[p1:abc][p2:123][p3:xxx]
result	返回结果
sqlIn	SQL语句
sqlParameter	SQL参数
sqlOut	SQL结果
exception	错误信息

4. Zookeeper搭建：

4.1. 修改配置文件：

$cd zookeeper														
$cd conf														
$cp zoo_sample.cfg zoo.cfg														
$vi zoo.cfg														
	   tickTime=2000     默认值													
	   initLimit=10      默认值													
	   syncLimit=5       默认值													
	   dataDir=/home/hadoop/zookeeper  自己创建的zookeeper目录													
	   clientPort=2181    默认值

4.2. 启动ZooKeeper：

[hadoop@hadoop bin]$ sh zkServer.sh start	
ZooKeeper JMX enabled by default	
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg	
Starting zookeeper ... STARTED

4.3. 验证：

$ jps	
22496 NameNode	
15520 QuorumPeerMain	
15921 Jps	
22628 DataNode	
29349 Main	
23111 NodeManager	
28648 Main	
30393 Main	
2141 Application	
23006 ResourceManager	
22830 SecondaryNameNode

5. HBase：

5.1. 修改hbase-env.sh

export JAVA_HOME=/usr/lib/java/jdk1.8.0_172		
export HBASE_MANAGES_ZK=false		
↑使用内部zookeeper和外部zookeeper的唯一区别，其他hbase-site.xml的配置还是需要的

5.2. 修改hbase-site.xml

<configuration>											
    <property>											
        <name>hbase.rootdir</name>											
        <value>hdfs://hadoop:9000/hbase</value>											
    </property>											
    <property>											
        <name>hbase.cluster.distributed</name>											
        <value>true</value>											
    </property>											
    <property>											
        <name>hbase.zookeeper.quorum</name>											
        <value>hadoop</value>											
    </property>											
    <property>											
        <name>hbase.temp.dir</name>											
        <value>/home/hadoop/hbase/data/tmp</value>											
    </property>											
    <property>											
        <name>hbase.zookeeper.property.clientPort</name>											
        <value>2181</value>											
    </property>											
    <property>											
        <name>hbase.master.info.port</name>											
        <value>60000</value>											
    </property>											
</configuration>

5.3. Hbase启动：

$start-hbase.sh

5.4. 验证：

[hadoop@hadoop bin]$ jps														
11607 ResourceManager														
11447 SecondaryNameNode														
17224 HRegionServer														
11273 DataNode														
13866 Application														
17099 HMaster														
11708 NodeManager														
11148 NameNode														
17805 Jps														
16926 QuorumPeerMain														
														
[hadoop@hadoop bin]$ hbase shell														
SLF4J: Class path contains multiple SLF4J bindings.														
SLF4J: Found binding in [jar:file:/usr/local/sun/hbase/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]														
SLF4J: Found binding in [jar:file:/usr/local/sun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]														
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.														
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]														
HBase Shell														
Use "help" to get list of supported commands.														
Use "exit" to quit this interactive shell.														
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell														
Version 2.0.4, r205e39c5704bf38568b34926dde9f1ee76e6b5d0, Fri Dec 28 22:13:42 PST 2018														
Took 0.0034 seconds                                                                                                                                                                            														
hbase(main):001:0> list														
TABLE                                                                                                                                                                                          														
0 row(s)														
Took 0.4661 seconds                                                                                                                                                                            														
=> []														
hbase(main):002:0> [hadoop@hadoop bin]$ hadoop fs -ls /														
Found 2 items														
drwxr-xr-x   - root   supergroup          0 2019-01-07 14:20 /hbase														
drwxr-xr-x   - hadoop supergroup          0 2019-01-07 14:10 /smphbeatslog

四. 通过log4j+Flume将日志写入HDFS：

我这里写日志的时候还是做了一下格式的统一，方便以后的日志分析

其实主要是通过拦截器和过滤器，获取接口调用时的接口和参数，以及SQL调用时的SQL语句和参数。

1. 在web.xml中添加过滤器，获取接口返回时的信息

	<!-- Response filter-->
	<filter>
		<filter-name>ResponseFilter</filter-name>
		<filter-class>com.thinkgem.jeesite.common.logService.LogFilter</filter-class>
	</filter>
	<filter-mapping>
		<filter-name>ResponseFilter</filter-name>
		<url-pattern>/*</url-pattern>
	</filter-mapping>

2. 在spring-mvc.xml中，添加拦截器，获取在接口调用前的接口信息

	<mvc:interceptors>
		<mvc:interceptor>
			<mvc:mapping path="/**" />
			<bean class="com.thinkgem.jeesite.common.logService.LogInterceptor" />
		</mvc:interceptor>
	</mvc:interceptors>

3. 在mybatis-config.xml中添加插件，获取SQL执行时的信息

	<!-- 插件配置 -->
	<plugins>
		<plugin interceptor="com.thinkgem.jeesite.common.logService.SqlLogInterceptor" />
    </plugins>

4. log4j的配置如下：

只写入WARN级别的原因是，很多系统自带的INFO日志没什么用，所以提高到WARN，可以减少不必要的日志。

log4j.rootLogger=INFO,stdout,flume

log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target = System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 122.144.211.06
log4j.appender.flume.Port = 55555
log4j.appender.flume.UnsafeMode = true
log4j.appender.flume.Threshold = WARN
log4j.appender.flume.layout=org.apache.log4j.PatternLayout

5. 定义LogEntity.java作为一条日志的对象：

package com.thinkgem.jeesite.common.logService;

import org.apache.commons.lang3.StringUtils;

import java.text.Format;
import java.text.SimpleDateFormat;
import java.util.Date;

public class LogEntity {

	// 基本信息
	private String id;
	
	private String format="0";		// 日志格式的版本
    private String app;            // 调用类型 如：[app:ExamCenter]
    private String type="interfaceIn";	// 类型 如：[type:sql][type:interfaceIn][type:interfaceOut][type:exception]
    private String userAgent;		// 客户端
    private String userId;          // 用户ID
    private String userName;        // 用户名
    private String insertDate;      // 调用日期
    
    // 请求信息
    private String domain;       	// 域名
    private String interFace;       // 接口
    private String interFaceParam; // 接口参数
    
    // 返回结果
    private String success;          // 1:调用成功 0：调用失败
    private String message;         // 调用信息
    private String parameter;       // 用户自定义参数：[p1:abc][p2:123][p3:xxx]
    private String result;			// 返回结果
    
    // SQL
    private String sqlIn;			// SQL语句
    private String sqlParameter;	// SQL参数
    private String sqlOut;			// SQL结果
    
    // 错误信息
    private String exception;		// 错误信息

    public String getUserId() {
        return userId;
    }

    public void setUserId(String userId) {
        this.userId = userId;
    }

    public String getUserName() {
        return userName;
    }

    public void setUserName(String userName) {
        this.userName = userName;
    }

    public String getInterFace() {
        return interFace;
    }

    public void setInterFace(String interFace) {
        this.interFace = interFace;
    }

    public String getInterFaceParam() {
        return interFaceParam;
    }

    public void setInterFaceParam(String interFaceParam) {
        this.interFaceParam = interFaceParam;
    }

    public String getInsertDate() {
        if (StringUtils.isBlank(insertDate)){
            Format format = new SimpleDateFormat("yyyy-MM-dd HH-mm-ss");
            insertDate = format.format(new Date());
        }

        return insertDate;
    }

    public void setInsertDate(String insertDate) {
        this.insertDate = insertDate;
    }

    public String getParameter() {
        return parameter;
    }

    public void setParameter(String parameter) {
        this.parameter = parameter;
    }

    public String getSuccess() {
        if (StringUtils.isBlank(success))
            success = "0";

        return success;
    }

    public void setSuccess(String success) {
        this.success = success;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

    public String getApp() {
        if (StringUtils.isBlank(app))
            app = "ExamCenter";

        return app;
    }

    public void setApp(String app) {
        this.app = app;
    }

	public String getFormat() {
		return format;
	}

	public void setFormat(String format) {
		this.format = format;
	}

	public String getResult() {
		return result;
	}

	public void setResult(String result) {
		this.result = result;
	}

	public String getType() {
		return type;
	}

	public void setType(String type) {
		this.type = type;
	}

	public String getDomain() {
		return domain;
	}

	public void setDomain(String domain) {
		this.domain = domain;
	}

	public String getException() {
		return exception;
	}

	public void setException(String exception) {
		this.exception = exception;
	}

	public String getUserAgent() {
		return userAgent;
	}

	public void setUserAgent(String userAgent) {
		this.userAgent = userAgent;
	}

	public String getSqlIn() {
		return sqlIn;
	}

	public void setSqlIn(String sqlIn) {
		this.sqlIn = sqlIn;
	}

	public String getSqlOut() {
		return sqlOut;
	}

	public void setSqlOut(String sqlOut) {
		this.sqlOut = sqlOut;
	}

	public String getSqlParameter() {
		return sqlParameter;
	}

	public void setSqlParameter(String sqlParameter) {
		this.sqlParameter = sqlParameter;
	}

	public String getId() {
		return id;
	}

	public void setId(String id) {
		this.id = id;
	}
}

6. web.xml中用到的LogFilter.java：

package com.thinkgem.jeesite.common.logService;

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.alibaba.fastjson.JSONObject;
import com.inesa.common.utils.StringUtils;
import com.thinkgem.jeesite.modules.sys.entity.User;
import com.thinkgem.jeesite.modules.sys.utils.UserUtils;


public class LogFilter implements Filter {

    @Override
    public void destroy() {
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse resp, FilterChain chain)
            throws IOException, ServletException {

        resp.setContentType("text/html;charset=utf-8");
        HttpServletResponse response = (HttpServletResponse) resp;

        ResponseReplaceWrapper responseReplaceWrapper = new ResponseReplaceWrapper(response);
        chain.doFilter(request, responseReplaceWrapper);

        String out = responseReplaceWrapper.getCharWriter().toString();

        PrintWriter printWriter = response.getWriter();
        printWriter.write(out);
        
        // ---------------------------------log save-------------------------------------
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        LogEntity log = new LogEntity();
        
        // type
        log.setType("interfaceOut");
        
        // userId
        // userName
        User user = UserUtils.getUser();
        if (user != null && !StringUtils.isBlank(user.getId())) {
            log.setUserId(user.getId());
            log.setUserName(user.getName());
        } else {
            log.setUserId("");
            log.setUserName("");
        }
        
        // domain
        StringBuffer url = httpRequest.getRequestURL();  
        String domainUrl = url.delete(url.length() - httpRequest.getRequestURI().length(), url.length()).append("/").toString();  
        log.setDomain(domainUrl);
        
        // interFace
        if (StringUtils.isBlank(httpRequest.getQueryString())){
            log.setInterFace(httpRequest.getRequestURI());
        }else {
            log.setInterFace(httpRequest.getRequestURI() + "?" + httpRequest.getQueryString());
        }
        
        // interFaceParam        
        log.setInterFaceParam(JSONObject.toJSONString(httpRequest.getParameterMap()));
        
        // userAgent
        log.setUserAgent(httpRequest.getHeader("user-agent"));

        if (!StringUtils.isBlank(out)){
        	if (getContent(out, "result").equals("Success"))
        		log.setSuccess("1");
        	else
        		log.setSuccess("0");
	        log.setMessage(getContent(out, "message"));
	        // stackoverflow可能
	        //log.setResult(out);
        }
        
        LogUtils.saveLog(log);
        // ------------------------------------log save----------------------------------
    }

    @Override
    public void init(FilterConfig filterConfig) throws ServletException {
    }

    public static String getContent(String info, String property){
		String result = "";
		
		int intStart = info.indexOf(property);
		if (intStart > 0){
			int intEnd = info.indexOf(",", intStart);
			
			result = info.substring(intStart+property.length()+3, intEnd-1);
		}
			
		return result;
	}
}

7. spring-mvc.xml中用到的LogInterceptor：

package com.thinkgem.jeesite.common.logService;

import java.text.SimpleDateFormat;
import java.util.Date;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.springframework.core.NamedThreadLocal;
import org.springframework.web.servlet.HandlerInterceptor;
import org.springframework.web.servlet.ModelAndView;

/**
 * 日志拦截器
 */
public class LogInterceptor extends BaseService implements HandlerInterceptor {

	private static final ThreadLocal<Long> startTimeThreadLocal =
			new NamedThreadLocal<Long>("ThreadLocal StartTime");
	
	@Override
	public boolean preHandle(HttpServletRequest request, HttpServletResponse response, 
			Object handler) throws Exception {
		LogUtils.writeLogBefore(request);
		return true;
	}

	@Override
	public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, 
			ModelAndView modelAndView) throws Exception {
		if (modelAndView != null){
			logger.info("ViewName: " + modelAndView.getViewName());
		}
	}

	@Override
	public void afterCompletion(HttpServletRequest request, HttpServletResponse response, 
			Object handler, Exception ex) throws Exception {

		// 保存日志
		LogUtils.writeLogError(request, response, handler, ex);		
	}

}

8. mybatis-config.xml中用到的SqlLogInterceptor：

package com.thinkgem.jeesite.common.logService;

import org.apache.ibatis.executor.Executor;
import org.apache.ibatis.mapping.BoundSql;
import org.apache.ibatis.mapping.MappedStatement;
import org.apache.ibatis.mapping.SqlSource;
import org.apache.ibatis.plugin.Intercepts;
import org.apache.ibatis.plugin.Invocation;
import org.apache.ibatis.plugin.Plugin;
import org.apache.ibatis.plugin.Signature;
import org.apache.ibatis.reflection.MetaObject;
import org.apache.ibatis.session.ResultHandler;
import org.apache.ibatis.session.RowBounds;

import java.util.Properties;

/**
 * 数据库日志保存
 * @author sun
 * @version 2019-1-3
 */
@Intercepts({
	@Signature(type = Executor.class, method = "update", 
			args = {MappedStatement.class, Object.class }),  
    @Signature(type = Executor.class, method = "query",
    		args = {MappedStatement.class, Object.class, RowBounds.class, ResultHandler.class})
	})
public class SqlLogInterceptor extends BaseInterceptor {

    private static final long serialVersionUID = 1L;

    @Override
    public Object intercept(Invocation invocation) throws Throwable {

        final MappedStatement mappedStatement = (MappedStatement) invocation.getArgs()[0];
        
        Object parameter = invocation.getArgs()[1];
        BoundSql boundSql = mappedStatement.getBoundSql(parameter);
    	
    	LogUtils.writeLogSQL(StringUtils.replaceEach(boundSql.getSql().toString(), new String[]{"\n","\t"}, new String[]{" "," "}), parameter.toString(), "");
	        
        return invocation.proceed();
    }


    @Override
    public Object plugin(Object target) {
        return Plugin.wrap(target, this);
    }

    @Override
    public void setProperties(Properties properties) {
        super.initProperties(properties);
    }
}

9. ResponseReplaceWrapper用来获取接口返回时的信息：

package com.thinkgem.jeesite.common.logService;

import java.io.CharArrayWriter;
import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletResponseWrapper;

public class ResponseReplaceWrapper extends HttpServletResponseWrapper {

    private CharArrayWriter charArrayWriter=new CharArrayWriter();

    public ResponseReplaceWrapper(HttpServletResponse response) {
        super(response);
    }

    @Override
    public PrintWriter getWriter() throws IOException {
        return new PrintWriter(charArrayWriter);
    }

    public CharArrayWriter getCharWriter(){
        return charArrayWriter;
    }
}

10. 统一异常处理GlobalExceptionHandler：

package com.thinkgem.jeesite.common.logService;

import com.alibaba.fastjson.JSONObject;
import com.thinkgem.jeesite.common.utils.Exceptions;
import com.thinkgem.jeesite.common.utils.StringUtils;
import com.thinkgem.jeesite.modules.bt.util.DateUtil;
import com.thinkgem.jeesite.modules.sys.entity.User;
import com.thinkgem.jeesite.modules.sys.utils.UserUtils;
import org.apache.shiro.authc.AuthenticationException;
import org.springframework.validation.BindException;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;

import javax.servlet.http.HttpServletRequest;
import javax.validation.ConstraintViolationException;
import javax.validation.ValidationException;
import java.util.Date;

/**
 * 统一异常处理
 */
@ControllerAdvice
public class GlobalExceptionHandler {


    /**
     * 应用到所有@RequestMapping注解的方法,在其抛出Exception时执行
     */
    @ExceptionHandler(value = Exception.class)
    public String jsonErrorHandler(HttpServletRequest request, Exception e) {
        saveLog(request,e);
        //参数绑定异常
        if (e.getClass().isInstance(BindException.class) || e.getClass().isInstance(ConstraintViolationException.class)
                || e.getClass().isInstance(ValidationException.class)){
            return "error/400";
        }
        //授权登录异常
        if (e.getClass().isInstance(AuthenticationException.class)){
            return "error/403";
        }
        return "error/400";
    }

    private void saveLog(HttpServletRequest request, Exception e){
        User user = UserUtils.getUser();
        LogEntity log = new LogEntity();
        if (user != null && !StringUtils.isBlank(user.getId())) {
            log.setUserId(user.getUserId());
        } else {
            log.setUserId("");
        }
        if (StringUtils.isBlank(request.getQueryString())){
            log.setInterFace(request.getRequestURI());
        }else {
            log.setInterFace(request.getRequestURI() + "?" + request.getQueryString());
        }
        log.setSuccess("0");
        log.setInsertDate(DateUtil.yyyyMMddHHmmssFormat(new Date()));
        log.setApp("smph_beats");
        log.setInterFaceParam(JSONObject.toJSONString(request.getParameterMap()));
        if (e != null) {
            log.setMessage(Exceptions.getStackTraceAsString(e));
        }
        com.thinkgem.jeesite.common.logService.LogUtils.saveLog(log);
    }
}

11. LogUtils.java用来根据指定格式存储日志：

package com.thinkgem.jeesite.common.logService;

import java.io.UnsupportedEncodingException;
import java.lang.reflect.Method;
import java.net.URLEncoder;
import java.util.Date;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.alibaba.fastjson.JSONObject;
import com.inesa.common.utils.StringUtils;
import com.thinkgem.jeesite.common.utils.Exceptions;
import com.thinkgem.jeesite.modules.bt.util.DateUtil;
import com.thinkgem.jeesite.modules.sys.entity.Log;
import com.thinkgem.jeesite.modules.sys.entity.User;
import com.thinkgem.jeesite.modules.sys.utils.UserUtils;

import org.apache.log4j.Logger;
import org.apache.shiro.authz.annotation.RequiresPermissions;
import org.springframework.web.method.HandlerMethod;

public class LogUtils {

	private static Logger logger = Logger.getLogger(LogUtils.class.getName());

	public static void writeLogBefore(HttpServletRequest request){
        LogEntity log = new LogEntity();
        
		// userId
		// userName
        User user = UserUtils.getUser();
        if (user != null && !StringUtils.isBlank(user.getId())) {
            log.setUserId(user.getId());
            log.setUserName(user.getName());
        } else {
            log.setUserId("");
            log.setUserName("");
        }
        
        // domain
        StringBuffer url = request.getRequestURL();  
        String domainUrl = url.delete(url.length() - request.getRequestURI().length(), url.length()).append("/").toString();  
        log.setDomain(domainUrl);
        
        // interFace
        if (StringUtils.isBlank(request.getQueryString())){
            log.setInterFace(request.getRequestURI());
        }else {
            log.setInterFace(request.getRequestURI() + "?" + request.getQueryString());
        }
        
        // interFaceParam
        log.setInterFaceParam(JSONObject.toJSONString(request.getParameterMap()));
        
        // success
        log.setSuccess("-");

        // userAgent
        log.setUserAgent(request.getHeader("user-agent"));
        
        saveLog(log);
	}
	
	/**
	 * 保存日志
	 */
	public static void writeLogError(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex){
		if (ex != null){
	        LogEntity log = new LogEntity();
	        
	        // type
	        log.setType("exception");
	        
	        // userId
	        // userName
	        User user = UserUtils.getUser();
	        if (user != null && !StringUtils.isBlank(user.getId())) {
	            log.setUserId(user.getId());
	            log.setUserName(user.getName());
	        } else {
	            log.setUserId("");
	            log.setUserName("");
	        }
	        
	        // domain
	        StringBuffer url = request.getRequestURL();  
	        String domainUrl = url.delete(url.length() - request.getRequestURI().length(), url.length()).append("/").toString();  
	        log.setDomain(domainUrl);
	        
	        // interFace
	        if (StringUtils.isBlank(request.getQueryString())){
	            log.setInterFace(request.getRequestURI());
	        }else {
	            log.setInterFace(request.getRequestURI() + "?" + request.getQueryString());
	        }
	        
	        // interFaceParam
	        log.setInterFaceParam(JSONObject.toJSONString(request.getParameterMap()));
	        
	        // userAgent
	        log.setUserAgent(request.getHeader("user-agent"));
	        
	        // exception
	        log.setException(Exceptions.getStackTraceAsString(ex));
	        
	        // 获取返回的数据
	        try {
		        log.setSuccess("0");
		        
			} catch (Exception e) {
				e.printStackTrace();
			}
	        
	        saveLog(log);
		}
	}
	
	/**
	 * 保存SQL语句
	 */
	public static void writeLogSQL(String sqlIn, String sqlParameter, String sqlOut){
        LogEntity log = new LogEntity();
        
        // type
        log.setType("sql");
        
        // userId
        // userName
        User user = UserUtils.getUser();
        if (user != null && !StringUtils.isBlank(user.getId())) {
            log.setUserId(user.getId());
            log.setUserName(user.getName());
        } else {
            log.setUserId("");
            log.setUserName("");
        }
        
        // SQL
        log.setSqlIn(sqlIn);
        log.setSqlParameter(sqlParameter);
        log.setSqlOut(sqlOut);
        
        // 获取返回的数据
        try {
	        log.setSuccess("-");
	        
		} catch (Exception e) {
			e.printStackTrace();
		}
        
        saveLog(log);
	}
	

	/**
	 * 保存日志线程
	 */
	public static class SaveLogThread extends Thread{
		
		private Log log;
		private Object handler;
		private Exception ex;
		
		public SaveLogThread(Log log, Object handler, Exception ex){
			super(SaveLogThread.class.getSimpleName());
			this.log = log;
			this.handler = handler;
			this.ex = ex;
		}
		
		@Override
		public void run() {
			// 获取日志标题
			if (StringUtils.isBlank(log.getTitle())){
				String permission = "";
				if (handler instanceof HandlerMethod){
					Method m = ((HandlerMethod)handler).getMethod();
					RequiresPermissions rp = m.getAnnotation(RequiresPermissions.class);
					permission = (rp != null ? StringUtils.join(rp.value(), ",") : "");
				}
				log.setTitle("???");
			}
			// 如果有异常，设置异常信息
			log.setException(Exceptions.getStackTraceAsString(ex));
		}
	}
	
	public static void saveLog(LogEntity logEntity){
		
		try {
			StringBuilder log = new StringBuilder();

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getFormat()))
				log.append("[format:1]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getApp()))
				log.append("[app:smph_beats]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getType()))
				log.append("[type:" + logEntity.getType() + "]");

			if (!StringUtils.isBlank(logEntity.getUserAgent()))
				log.append("[userAgent:" + logEntity.getUserAgent() + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getUserId()))
				log.append("[userId:" + logEntity.getUserId() + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getUserName())){
				log.append("[userName:" + URLEncoder.encode(logEntity.getUserName(), "UTF-8") + "]");
			}

			log.append("[insertDate:" + DateUtil.yyyyMMddHHmmssFormat(new Date()) + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getDomain()))
				log.append("[domain:" + logEntity.getDomain() + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getInterFace()))
				log.append("[interFace:" + logEntity.getInterFace() + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getInterFaceParam()))
				log.append("[interFaceParam:" + URLEncoder.encode(logEntity.getInterFaceParam(), "UTF-8") + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getSuccess()))
				log.append("[success:" + logEntity.getSuccess() + "]");

			if (!com.inesa.common.utils.StringUtils.isBlank(logEntity.getMessage()))
				log.append("[message:" + logEntity.getMessage() + "]");

			if (!StringUtils.isBlank(logEntity.getParameter()))
				log.append("[parameter:" + URLEncoder.encode(logEntity.getParameter(), "UTF-8") + "]");

			if (!StringUtils.isBlank(logEntity.getResult()))
				log.append("[result:" + logEntity.getResult() + "]");

			if (!StringUtils.isBlank(logEntity.getSqlIn()))
				log.append("[sqlIn:" + logEntity.getSqlIn() + "]");

			if (!StringUtils.isBlank(logEntity.getSqlParameter()))
				log.append("[sqlParameter:" + URLEncoder.encode(logEntity.getSqlParameter(), "UTF-8") + "]");

			if (!StringUtils.isBlank(logEntity.getSqlOut()))
				log.append("[sqlOut:" + logEntity.getSqlOut() + "]");

			if (!StringUtils.isBlank(logEntity.getException()))
				log.append("[exception:" + logEntity.getException() + "]");

			logger.warn(log.toString());

		} catch (Exception e) {
			System.out.println(e.getMessage());
		}
	}
}

至此，在项目中，调用的接口，执行的SQL，系统Exception等信息都可以写入日志了。

五. SQL Server表创建：

目前想要做的有两个：

1.错误信息追踪（SQL错误，接口错误）

2.用户接口调用日志

所以，根据上面的错误日志的格式定义，我觉得必须的检索条件有以下这些：

app	调用类型如：[app:ExamCenter]
type	日志类型如：[type:sql][type:interfaceIn][type:interfaceOut][type:exception]
userId	用户ID
userName	用户名
insertDate	调用日期
interFace	接口
interFaceParam	接口参数
success	1：成功 0：失败
parameter	用户自定义参数：[p1:abc][p2:123][p3:xxx]
sqlIn	SQL语句

而Hbase是Key-Value类型的数据库，它的row-key并不适合存放太多的信息，因此无法满足多条件检索。

所以我觉得还是将数据存放到SQL Server里面查询比较好。

1. 创建表(app)_log，字段同日志格式，每个app都创建不同的表。

列名	逻辑名	数据类型	允许Null	主键	自动增长	默认值	其他
id	ID	varchar(64)		○
format	日志格式的版本	varchar(64)				1
type	类型	varchar(64)					sql interfaceIn interfaceOut exception
userAgent	客户端	varchar(64)	○
userId	用户ID	varchar(64)	○
userName	用户名	varchar(64)	○
insertDate	调用日期	datetime	○
domain	域名	varchar(64)	○
interFace	接口	varchar(255)	○
interFaceParam	接口参数	varchar(2000)	○
success	调用结果	char(1)	○			'-'	1:调用成功 0：调用失败
message	返回message	varchar(2000)	○
parameter	app自定义参数	varchar(2000)	○
result	返回结果	varchar(Max)	○
sqlIn	SQL语句	varchar(5000)	○
sqlParameter	SQL参数	varchar(2000)	○
sqlOut	SQL结果	varchar(Max)	○
exception	错误信息	varchar(Max)	○
detail	日志细节	varchar(Max)
fileName	文件名	varchar(255)	○

2. 创建表file_read_log（日志文件读取履历表），用来记录已经读取的HDFS日志文件，避免重复加载

列名	逻辑名	数据类型	允许Null	主键	自动增长	默认值	其他
id	ID	varchar(64)		○
app	APP名	varchar(64)	○
service	数据统计服务名	varchar(64)	○
name	读取的最后的文件	varchar(255)	○
create_date	创建时间	datetime	○
update_date	更新时间	datetime	○

六. 使用Spark将数据写入SQL Server:

1. 本地调试时用：

1.1.准备下面的jar包

hadoop-2.9.1.tar.gz

spark-2.3.1-bin-hadoop2.7.tgz

hadoop2.9.0开发调试工具（文章最后有下载）

1.2. 配置环境变量

把上面的包都解压到D盘下

如：D:/hadoop-2.9.1

在环境变量中新建HADOOP_HOME->D:/hadoop-2.9.1

在环境变量中新建SPARK_HOME->D:/spark-2.3.1-bin-hadoop2.7

把%HADOOP_HOME%\bin;%SPARK_HOME%\bin;加入PATH中

1.3. 拷贝Hadoop-Common 下的文件

官方下载的Apache hadoop 2.6.4的压缩包里，缺少windows下运行的链接库（hadoop.dll，winutils.exe，libwinutils.lib等）。

下载Hadoop-Common后直接解压，把里面的文件全部拷贝到官方hadoop目录下的bin目录即可。

2. pom.xml中增加：

<!-- spark -->														
<dependency>														
   <groupId>jdk.tools</groupId>														
   <artifactId>jdk.tools</artifactId>														
   <version>1.8</version>														
   <scope>system</scope>														
   <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>														
</dependency>														
<dependency>														
   <groupId>org.apache.spark</groupId>														
   <artifactId>spark-core_2.10</artifactId>														
   <version>1.6.1</version>														
</dependency>														
      <dependency>														
          <groupId>com.sun.jersey</groupId>														
          <artifactId>jersey-server</artifactId>														
          <version>1.8</version>														
      </dependency>														
<!-- spark -->

3. 定时器（Java）：

package com.inesa.hadoop.controller;

import com.alibaba.druid.util.StringUtils;
import com.inesa.hadoop.entity.FileReadLogEntity;
import com.inesa.hadoop.entity.LogEntity;
import com.inesa.hadoop.service.FileReadLogService;
import com.inesa.hadoop.service.LogService;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import scala.Serializable;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.File;
import java.io.UnsupportedEncodingException;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.net.URI;
import java.net.URLDecoder;
import java.util.List;

/**
 * 定时器Controller
 * @author sun
 * @version 2019-01-09
 */
@Controller
@RequestMapping(value = "timer")
public class TimerController implements Serializable {

    @Autowired
    private LogService logService;

    @Autowired
    private FileReadLogService fileReadLogService;

    @Scheduled(cron = "0 * * * * *")    // 每隔1分钟执行一次
    public void hdfsToMssql() {
        System.out.println("hdfsToMssql Start:" + new java.sql.Timestamp(System.currentTimeMillis()).toString());

        SparkConf conf=new SparkConf().setMaster("local").setAppName("sunTest");
        JavaSparkContext sc =new JavaSparkContext(conf);

        try{
            //String mod = "test";
            String mod = "centos";

            if (mod.equals("test")){
                String path = "D:/tmp/";
                // 获得指定文件对象
                File file = new File(path);
                // 获得该文件夹内的所有文件
                File[] array = file.listFiles();
                for(int i=0;i<array.length;i++) {
                    //如果是文件夹
                    if(array[i].isDirectory()) {
                        if (!existTable(array[i].getName()))
                            createTable(array[i].getName());

                        // 获取此目录下的文件列表
                        String pathSub = array[i] + "/";
                        // 获得指定文件对象
                        File fileSub = new File(pathSub);
                        // 获得该文件夹内的所有文件
                        File[] arraySub = fileSub.listFiles();

                        // 获得文件读取履历
                        boolean blnRead = true;     // 是否能够读取此文件
                        FileReadLogEntity fileReadLog = new FileReadLogEntity();
                        fileReadLog.setApp(array[i].getName());
                        fileReadLog.setService("all");
                        List<FileReadLogEntity> logList = fileReadLogService.findList(fileReadLog);
                        if (logList.size()>0) {
                            fileReadLog = logList.get(0);
                            blnRead = false;
                        }

                        for(int j=0;j<arraySub.length;j++) {
                            //如果是文件
                            if(arraySub[j].isFile())
                            {
                                if (arraySub[j].getName().indexOf(".tmp")>0) {
                                    continue;
                                }

                                // 判断已读取过的日志的日期是否在当前文件之前
                                if (Long.parseLong(fileReadLog.getName().replace("FlumeData", "").replace(".txt", ""))
                                        < Long.parseLong(arraySub[j].getName().replace("FlumeData", "").replace(".txt", "")))
                                    blnRead = true;

                                // 如果可以读取
                                if (blnRead) {
                                    fileProcess(conf, sc, pathSub, arraySub[j].getName(), array[i].getName());
                                    //System.out.println(pathSub + arraySub[j].getName());

                                    // 插入文件读取履历
                                    fileReadLog.setName(arraySub[j].getName());
                                    fileReadLogService.save(fileReadLog);
                                }else{
                                    // 找到上次读取过的文件以后，下一个文件开始就可以进行读取
                                    if (fileReadLog.getName().equals(arraySub[j].getName()))
                                        blnRead = true;
                                }
                            }
                        }
                    }
                }
            }else if (mod.equals("centos")){
                String path = "hdfs://hadoop:9000/";
                Configuration configuration = new Configuration();
                FileSystem fs = FileSystem.get(URI.create(path), configuration);
                FileStatus[] status = fs.listStatus(new Path(path));
                for (FileStatus file : status) {
                    if (file.isDirectory()){
                        if (!existTable(file.getPath().getName()))
                            createTable(file.getPath().getName());

                        // 获取此目录下的文件列表
                        String pathSub = file.getPath() + "/";
                        // 获得指定文件对象
                        FileSystem fsSub = FileSystem.get(URI.create(pathSub), configuration);
                        // 获得该文件夹内的所有文件
                        FileStatus[] statusSub = fsSub.listStatus(new Path(pathSub));

                        // 获得文件读取履历
                        boolean blnRead = true;     // 是否能够读取此文件
                        FileReadLogEntity fileReadLog = new FileReadLogEntity();
                        fileReadLog.setApp(file.getPath().getName());
                        fileReadLog.setService("all");
                        List<FileReadLogEntity> logList = fileReadLogService.findList(fileReadLog);
                        if (logList.size()>0) {
                            fileReadLog = logList.get(0);
                            blnRead = false;
                        }

                        for (FileStatus fileSub : statusSub) {
                            //如果是文件
                            if(fileSub.isFile())
                            {
                                if (fileSub.getPath().getName().indexOf(".tmp")>0)
                                    continue;

                                if (StringUtils.isEmpty(fileSub.getPath().getName())
                                        || !StringUtils.isNumber(fileSub.getPath().getName().replace("FlumeData.", "")))
                                    continue;

                                // 判断已读取过的日志的日期是否在当前文件之前
                                if (Long.parseLong(fileSub.getPath().getName().replace("FlumeData.", ""))
                                        < Long.parseLong(fileSub.getPath().getName().replace("FlumeData.", "")))
                                    blnRead = true;

                                if (blnRead) {
                                    fileProcess(conf, sc, pathSub, fileSub.getPath().getName(), file.getPath().getName());

                                    // 插入文件读取履历
                                    fileReadLog.setName(fileSub.getPath().getName());
                                    fileReadLogService.save(fileReadLog);
                                }else{
                                    // 找到上次读取过的文件以后，下一个文件开始就可以进行读取
                                    if (fileReadLog.getName().equals(fileSub.getPath().getName()))
                                        blnRead = true;
                                }
                            }
                        }
                    }
                }
            }

            System.out.println("hdfsToMssql Success:" + new java.sql.Timestamp(System.currentTimeMillis()).toString());

        }catch(Exception e){
            System.out.println("hdfsToMssql Error:" + e.getMessage());
        }finally{
            sc.close();
        }
    }

    private void fileProcess(SparkConf conf, JavaSparkContext sc,String path, String fileName, String folderName) throws UnsupportedEncodingException {
        fileProcess(conf, sc, path, fileName, folderName, "");
    }

    private void fileProcess(SparkConf conf, JavaSparkContext sc,String path, String fileName, String folderName, String startName) throws UnsupportedEncodingException {

        JavaRDD<String> logData=sc.textFile(path + fileName).cache();

        for(String line:logData.collect()){
            if (!line.startsWith("[format:1]"))
                continue;

            LogEntity logEntity = new LogEntity();
            String[] lineArray = line.split("]\\[");
            for(int i=0;i<lineArray.length;i++){
                // 清理前后括号
                if(i==0)
                    lineArray[i] = lineArray[i].substring(1);
                else if (i==lineArray.length-1)
                    lineArray[i] = lineArray[i].substring(0, lineArray[i].length()-1);

                // 分离key和value
                int index = lineArray[i].indexOf(":");
                String key = lineArray[i].substring(0, index);
                key=key.substring(0,1).toUpperCase().concat(key.substring(1));
                String value = lineArray[i].substring(index+1);
                Field field = null;
                try {
                    if (key.equals("UserName")
                            || key.indexOf("param") > 0
                            || key.indexOf("Param") > 0){
                        Method m = logEntity.getClass().getDeclaredMethod("set" + key, String.class);
                        m.setAccessible(true);
                        value = value.replaceAll("%(?![0-9a-fA-F]{2})", "%25").replaceAll("\\+", "%2B");
                        m.invoke(logEntity, URLDecoder.decode(value, "UTF-8"));
                    }else {
                        Method m = logEntity.getClass().getDeclaredMethod("set" + key, String.class);
                        m.setAccessible(true);
                        m.invoke(logEntity, value);
                    }
                }catch (IllegalAccessException e) {
                    e.printStackTrace();
                } catch (NoSuchMethodException e) {
                    e.printStackTrace();
                } catch (InvocationTargetException e) {
                    e.printStackTrace();
                }
            }

            logEntity.setApp(folderName);
            logEntity.setFileName(fileName);
            line = line.replaceAll("%(?![0-9a-fA-F]{2})", "%25").replaceAll("\\+", "%2B");
            logEntity.setDetail(URLDecoder.decode(line, "UTF-8"));
            logService.save(logEntity);
        }
    }

    private void createTable(String folderName){
        LogEntity logEntity = new LogEntity();
        logEntity.setApp(folderName);
        logService.createTable(logEntity);
    }

    private boolean existTable(String folderName){
        LogEntity logEntity = new LogEntity();
        logEntity.setApp(folderName);
        List<LogEntity> tableList = logService.findTable(logEntity);
        if (tableList.size()>0)
            return true;
        else
            return false;
    }
}

其他的Service，Dao和Entity就不写了，dao.xml写一下吧

FileReadLogDao.xml：

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.inesa.hadoop.dao.FileReadLogDao">

	<sql id="fileReadLogColumns">
		a.id AS "id",
		a.app AS "app",
		a.service AS "service",
		a.name AS "name",
		a.create_date AS "createDate",
		a.update_date AS "updateDate"
	</sql>

	<sql id="fileReadLogJoins">
	</sql>

	<select id="findList" resultType="com.inesa.hadoop.entity.FileReadLogEntity">
		SELECT
			<include refid="fileReadLogColumns"/>
		FROM file_read_log a
		<where>
			1 = 1
			<if test="app != null and app != ''">
				AND a.app = #{app}
			</if>
			<if test="service != null and service != ''">
				AND a.service = #{service}
			</if>
		</where>
		ORDER BY a.create_date DESC
	</select>

    <insert id="insert">
		INSERT INTO file_read_log(
            id,
            app,
            service,
            name,
            create_date,
            update_date
		) VALUES (
            #{id},
            #{app},
            #{service},
            #{name},
            #{createDate},
            #{updateDate}
		)
	</insert>

	<update id="update">
		UPDATE file_read_log
		SET name = #{name},
			update_date = #{updateDate}
		WHERE id = #{id}
	</update>
</mapper>

七. 对外日志查询接口提供：

用Idea+SpringBoot，提供接口。

@RestController														
@RequestMapping(value = "log")														
public class LogController {														
														
   @Autowired														
   private LogService logService;														
														
    @RequestMapping(value = "search")														
   public void search(HttpServletRequest request, HttpServletResponse response,														
                  @RequestBody LogEntity logEntity) {														
      RestfulResult restfulResult = new RestfulResult();														
														
      try {														
         PageHelper.startPage(logEntity.getPageNo(), logEntity.getPageSize());														
														
         List<LogEntity> logList = logService.findList(logEntity);														
														
         PageInfo<LogEntity> pageInfo = new PageInfo<LogEntity>(logList);														
														
         restfulResult.setData(logList);														
         restfulResult.setCntData(pageInfo.getTotal());														
														
      } catch (Exception e) {														
         restfulResult.setResult("Error");														
         restfulResult.setMessage(e.getMessage());														
      }														
														
      CommUtils.printDataJason(response, restfulResult);														
   }														
}

LogDao.xml：

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.inesa.hadoop.dao.LogDao">

	<sql id="logColumns">
		a.id AS "id",
		a.format AS "format",
		a.type AS "type",
		a.userAgent AS "userAgent",
		a.userId AS "userId",
		a.userName AS "userName",
		a.insertDate AS "insertDate",
		a.domain AS "domain",
		a.interFace AS "interFace",
		a.interFaceParam AS "interFaceParam",
		a.success AS "success",
		a.message AS "message",
		a.parameter AS "parameter",
		a.result AS "result",
		a.sqlIn AS "sqlIn",
		a.sqlParameter AS "sqlParameter",
		a.sqlOut AS "sqlOut",
		a.exception AS "exception",
		a.detail AS "detail",
		a.fileName AS "fileName"
	</sql>

	<sql id="logJoins">
	</sql>

	<select id="findList" resultType="com.inesa.hadoop.entity.LogEntity">
		SELECT
			<include refid="logColumns"/>
		FROM ${app} a
		<where>
			1 = 1
			<if test="type != null and type != ''">
				AND a.type = #{type}
			</if>
			<if test="userId != null and userId != ''">
				AND a.userId = #{userId}
			</if>
			<if test="userName != null and userName != ''">
				AND a.userName LIKE '%'+#{userName}+'%'
			</if>
			<if test="insertDate != null and insertDate != ''">
				AND a.insertDate = #{insertDate}
			</if>
			<if test="startDate != null and startDate != ''">
				AND <![CDATA[a.insertDate >= #{startDate}]]>
			</if>
			<if test="endDate != null and endDate != ''">
				AND <![CDATA[a.insertDate <= #{endDate}]]>
			</if>
			<if test="interFace != null and interFace != ''">
				AND a.interFace LIKE '%'+#{interFace}+'%'
			</if>
			<if test="interFaceParam != null and interFaceParam != ''">
				AND a.interFaceParam LIKE '%'+#{interFaceParam}+'%'
			</if>
			<if test="success != null and success != ''">
				AND a.success = #{success}
			</if>
			<if test="parameter != null and parameter != ''">
				AND a.parameter LIKE '%'+#{parameter}+'%'
			</if>
			<if test="sqlIn != null and sqlIn != ''">
				AND a.sqlIn LIKE '%'+#{sqlIn}+'%'
			</if>
			<if test="sqlParameter != null and sqlParameter != ''">
				AND a.sqlParameter LIKE '%'+#{sqlParameter}+'%'
			</if>
		</where>
		ORDER BY a.insertDate DESC
	</select>

    <insert id="insert">
		INSERT INTO ${app}(
            id,
            format,
            type,
            userAgent,
            userId,
            userName,
            insertDate,
            domain,
            interFace,
            interFaceParam,
            success,
            message,
            parameter,
            result,
            sqlIn,
            sqlParameter,
            sqlOut,
            exception,
            detail,
            fileName
		) VALUES (
            #{id},
            #{format},
            #{type},
            #{userAgent},
            #{userId},
            #{userName},
            #{insertDate},
            #{domain},
            #{interFace},
            #{interFaceParam},
            #{success},
            #{message},
            #{parameter},
            #{result},
            #{sqlIn},
            #{sqlParameter},
            #{sqlOut},
            #{exception},
            #{detail},
            #{fileName}
		)
	</insert>

	<update id="createTable" parameterType="com.inesa.hadoop.entity.LogEntity">
      create table ${app}(
        [id] varchar(64) NOT NULL,
        [format] varchar(64) DEFAULT 1 NOT NULL,
        [type] varchar(64) NOT NULL,
        [userAgent] varchar(500) NULL,
        [userId] varchar(64) NULL,
        [userName] varchar(64) NULL,
        [insertDate] datetime NULL,
        [domain] varchar(255) NULL,
        [interFace] varchar(500) NULL,
        [interFaceParam] varchar(max) NULL,
        [success] char(1) DEFAULT '-' NULL,
        [message] varchar(max) NULL,
        [parameter] varchar(max) NULL,
        [result] varchar(max) NULL,
        [sqlIn] varchar(max) NULL,
        [sqlParameter] varchar(max) NULL,
        [sqlOut] varchar(max) NULL,
        [exception] varchar(max) NULL,
        [detail] varchar(max) NULL,
        [fileName] varchar(255) NULL,
        PRIMARY KEY (id))
    </update>

	<select id="findTable" resultType="com.inesa.hadoop.entity.LogEntity">
		SELECT id
		FROM [sysobjects]
		WHERE name = #{app}
	</select>
</mapper>

检索条件：

app	调用类型	smphbeats_log
type	日志类型	sql/interfaceIn/interfaceOut/exception
userId	用户ID
userName	用户名	like
insertDate	调用日期
startDate	开始日期
endDate	结束日期
interFace	接口	like
interFaceParam	接口参数	like
success	调用结果	1：成功 0：失败 -：未开始
parameter	用户自定义参数	like
sqlIn	SQL语句	like

验证：

八. 在CentOS7上部署logService服务：

1. 执行Maven下面的package

2. 用SSH Transfer工具，将target目录下生成的logService.jar复制到CentOS服务器上的/usr/local/sun/logService目录下

3. 验证启动：

#java -jar logService.jar

4. 注册为服务：

#cd /etc/systemd/system														
#vi logService.service														
输入：														
	[Unit]													
	Description=logService													
	After=syslog.target													
	[Service]													
	Type=simple													
	ExecStart=/usr/lib/java/jdk1.8.0_172/bin/java -jar /usr/local/sun/logService/logService.jar													
	[Install]													
	WantedBy=multi-user.target													
														
操作：														
	启动服务：				systemctl start serviceName									
	停止服务：				systemctl stop serviceName									
	服务状态：				systemctl status serviceName									
	项目日志：				journalctl -u serviceName									
	开机启动：				systemctl enable serviceName

5. 验证：