线上异常排查汇总【2】（持续更新）

1、redirect导致https变成http
最近线上碰见一个问题，应用使用了https，但是页面还是被拦截了，通过抓包发现中间有一段重定向变成了http。

根据参考链接：https://blog.youkuaiyun.com/zhuye1992/article/details/80496151

经过测试，经过如下，浏览器https请求到WAF（防火墙+负载均衡），WAF转发到应用服务器就变成http了。这里是因为nginx做转发的时候，没有配置成使用https转发。
然后代码中直接使用“redirect:”做重定向，然后前端范围的时候就变成http请求了，直到前端自发的页面跳转，ajax请求等操作后再变成https，如果有redirect操作，中间有一段时间还是http。

protocol:HTTP/1.0,uri:/XXXX,url:http://domain/XXXX

redirect的时候使用转发http是因为viewResolver在做重定向的时候，在redirectView文件中，参考链接中的解决方案实际都是为了不走到sendRedirect中，不走sendRedirect是返回相对路径，走sendRedirect是取请求的scheme，组成绝对路径返回，scheme中的协议是nginx中转发的时候的http，而不是https。请求是转发的http请求，所以有上面的问题。

2、Cannot call sendRedirect() after the response has been committed异常
错误日志

ERROR [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]] [DirectJDKLog.java:182] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: Cannot call sendRedirect() after the response has been committed] with root cause [] 
java.lang.IllegalStateException: Cannot call sendRedirect() after the response has been committed

原因：
在下载文件的时候，通过流的形式将内容输出到responseBody，流关闭的时候，responseBody的流貌似就不能再打开了，这个应该就是committed状态吧。
然后在方法的最后return XXX，一个string对象，spring会调用sendRedirect方法做重定向，这时候就报错了。
解决：
return 或者return null

2019-2-22更新
我改了之后，出现问题的次数大幅下降，但是还是出现了这个问题，迫使我只能去debug。
问题出现的原因有2点：
1、第一点就是上面的原因，因为返回了内容，只要改成void 方法或者return null就好了
2、出现了异常，具体异常流程如下：
下载文件的时候，下载时间过长（测试环境通过断点，一直停住），在往response中写数据的时候，发现链接断开了，出现了org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe的异常，虽然有try-catch，但是只捕获了flush的异常，在finally里，close的时候又出现了异常，也就是broken pipe的异常，父类中有@ExceptionHandler(Exception.class)注解，catch了这个异常，然后返回了return “redirect:/error.html”;
又回到了第一点问题。

@RequestMapping(value = "/downloadImage")
    @ResponseBody
    public String downloadImage(@RequestParam("key") String key, HttpServletResponse response) throws IOException {
        ServletOutputStream out = null;
        InputStream ips = null;
        try{
            if(StringUtils.isBlank(key)){
                return null;
            }
            ips=downloadFile(key);
            response.setContentType("multipart/form-data");
            response.setHeader("Content-disposition", "attachment; filename=" +key.substring(key.lastIndexOf("/"),key.length()));

            out = response.getOutputStream();
            //读取文件流
            int len = 0;
            byte[] buffer = new byte[1024 * 10];
            while ((len = ips.read(buffer)) != -1){
                out.write(buffer,0,len);
            }
            out.flush();
        }catch (Exception e){
            log.error("downloadImage exception:{}",e);
        }finally {
            out.close();
            ips.close();
        }
        return null;
    }

然后改成try-with-resource，由jdk处理异常，就可以了

try(ServletOutputStream out = response.getOutputStream();InputStream ips = fileOperationManager.downloadFile(key)){
            if(StringUtils.isBlank(key)){
                return null;
            }
            response.setContentType("multipart/form-data");
            response.setHeader("Content-disposition", "attachment; filename=" +key.substring(key.lastIndexOf("/"),key.length()));
            //读取文件流
            int len = 0;
            byte[] buffer = new byte[1024 * 10];
            while ((len = ips.read(buffer)) != -1){
                out.write(buffer,0,len);
            }
            out.flush();
        }catch (Exception e){
            log.error("downloadImage exception:{}",e);
        }

3、springboot无法启动
问题描述：springboot应用无法启动，报错日志如下。内部tomcat无法启动。

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. [] 
2019-04-15 18:15:19,108 [main] ERROR [org.springframework.boot.SpringApplication] [SpringApplication.java:842] - Application run failed [] 
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:155)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
	at com.XXX.XXX.XXX.Application.main(Application.java:27)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:126)
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86)
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:413)
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:179)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:152)
	... 16 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardServer[-1]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.startup.Tomcat.start(Tomcat.java:367)
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:107)
	... 21 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardService[Tomcat]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:793)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 23 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Tomcat]]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 25 common frames omitted
Caused by: org.apache.catalina.LifecycleException: A child container failed during start
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:949)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 27 common frames omitted

网上查了一下资料，说是项目中的servlet-api版本冲突了。
大家看下不一定是自己应用，可能是二方包，三方包引用的，看下引用依赖树，排除一下就可以了。

+- com.xuxueli:xxl-job-core:jar:1.9.2-SNAPSHOT:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] |  +- javax.servlet.jsp:jsp-api:jar:2.2:compile

com.XXX.share:jar:1.0.20:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  +- com.belerweb:pinyin4j:jar:2.5.0:compile

其中servlet.jar 是servlet 3.0 版本之前的地址，javax.servlet-api.jar 是servlet 3.0 版本之后的地址

**4、Transaction rolled back because it has been marked as rollback-only **
这个异常一般是一个大事务中包含多个子事务方法，因为传播级别的原因，共用一个事务，子事务抛出异常标记回滚，但是外层事务因为catch这个异常，正常commit。
具体可以参见：RCA: Spring 事务 rollback-only异常

如果子事务方法中try-catch，整体子事务方法没有抛出异常，是不会出现这个问题的，事务管理器会认为是无异常，正常commit

@Transactional(rollbackFor = Exception.class)
 public void subFunc(){
	throw new BizException()
}

@Transactional(rollbackFor = Exception.class)
 public void parentFunc(){
 	try{
 		subFunc();
 	}catch(Exception){
		log.error(e)
	}
}

这种是会抛出rollback-only异常

@Transactional(rollbackFor = Exception.class)
 public void subFunc(){
 	try{
 		throw new BizException()
 	}catch(Exception e){
		log.error(e);
	}
}

@Transactional(rollbackFor = Exception.class)
 public void parentFunc(){
	subFunc();
}

这种子事务方法内部catch了异常，子方法整体没有抛出异常，不会抛出rollback-only异常。因为这个异常没有被Transaction注解捕捉到。

TransactionAspectSupport#invokeWithinTransaction

.......
if (txAttr == null || !(tm instanceof CallbackPreferringPlatformTransactionManager)) {
			// Standard transaction demarcation with getTransaction and commit/rollback calls.
			TransactionInfo txInfo = createTransactionIfNecessary(tm, txAttr, joinpointIdentification);
			Object retVal = null;
			try {
				// This is an around advice: Invoke the next interceptor in the chain.
				// This will normally result in a target object being invoked.
				retVal = invocation.proceedWithInvocation();
			}
			catch (Throwable ex) {
				// target invocation exception
				completeTransactionAfterThrowing(txInfo, ex); //异常回滚，或者标记全局回滚。多个Transaction注解层层嵌套就会多次进入这个地方。直到某一层catch了异常，后面走了正常的commit逻辑，抛出rollback-only异常
				throw ex;
			}
			finally {
				cleanupTransactionInfo(txInfo);
			}
			commitTransactionAfterReturning(txInfo);
			return retVal;
		}
		.......