Web开发 之HttpClient详解

参考网址

HttpClient中文官网
HttpClient Apache官方文档
HttpClient 4.3教程
一个非常好的HttpClient系列教程

Maven依赖

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.10</version>
</dependency>
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.62</version>
</dependency>

HttpClient模拟post请求示例

使用HttpClient发送请求、接收响应一般步骤：

• 创建HttpClient对象。
• 创建请求方法的实例，并指定请求URL。如果需要发送GET请求，创建HttpGet对象；如果需要发送POST请求，创建HttpPost对象。
• 如果需要发送请求参数，可调用HttpGet、HttpPost共同的setParams(HttpParams params)方法来添加请求参数；对于HttpPost对象而言，也可调用setEntity(HttpEntity entity)方法来设置请求参数。
• 调用HttpClient对象的execute(HttpUriRequest request)发送请求，该方法返回一个HttpResponse。
• 调用HttpResponse的getAllHeaders()、getHeaders(String name)等方法可获取服务器的响应头；调用HttpResponse的getEntity()方法可获取HttpEntity对象，该对象包装了服务器的响应内容。程序可通过该对象获取服务器的响应内容。
• 释放连接。无论执行方法是否成功，都必须释放连接。

import org.apache.http.HttpEntity;
import org.apache.http.NameValuePair;
import org.apache.http.ParseException;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

public class SimpleHttpClientDemo {
    /**
     * 模拟请求
     *
     * @param url      资源地址
     * @param map      参数列表
     * @param encoding 编码
     * @return
     * @throws ParseException
     * @throws IOException
     */
    public static String send(String url, Map<String, String> map, String encoding) throws ParseException, IOException {
        String body = "";

        //创建httpclient对象
        CloseableHttpClient client = HttpClients.createDefault();
        //创建post方式请求对象
        HttpPost httpPost = new HttpPost(url);

        //装填参数
        List<NameValuePair> nvps = new ArrayList<NameValuePair>();
        if (map != null) {
            for (Map.Entry<String, String> entry : map.entrySet()) {
                nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));
            }
        }
        //设置参数到请求对象中
        httpPost.setEntity(new UrlEncodedFormEntity(nvps, encoding));

        System.out.println("请求地址：" + url);
        System.out.println("请求参数：" + nvps.toString());

        //设置header信息
        //指定报文头【Content-type】、【User-Agent】
        httpPost.setHeader("Content-type", "application/x-www-form-urlencoded");
        httpPost.setHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");

        //执行请求操作，并拿到结果（同步阻塞）
        CloseableHttpResponse response = client.execute(httpPost);
        //获取结果实体
        HttpEntity entity = response.getEntity();
        if (entity != null) {
            //按指定编码转换结果实体为String类型
            body = EntityUtils.toString(entity, encoding);
        }
        EntityUtils.consume(entity);
        //释放链接
        response.close();
        return body;
    }

}

HttpClient连接池

HttpClient连接池设置引发的一次雪崩
httpclient连接池的测试报告(jmeter做并发测试工具)
HttpClient高并发下性能优化
HttpClient连接池的一些思考
httpclient连接池是否能真的提高性能
Http持久连接与HttpClient连接池
HttpClient连接池配置

PoolingHttpClientConnectionManager pool = new PoolingHttpClientConnectionManager();
// 设置连接池的最大连接数
pool.setMaxTotal(200);
// 设置每个路由的最大连接数，路由是对MaxTotal的细分
pool.setDefaultMaxPerRoute(10);
// 在从连接池获取连接时，连接不活跃多长时间后需要进行一次验证，默认2s
pool.setValidateAfterInactivity(5*1000);
// 单独为某个站点设置最大连接个数
pool.setMaxPerRoute(new HttpRoute(new HttpHost("127.0.0.1", 80)), 50);
//默认请求配置
RequestConfig requestConfig = RequestConfig.custom()
        .setConnectTimeout(2*1000)// 设置连接超时时间 2s
        .setSocketTimeout(5*1000)// 设置等待数据超时时间 5s
        .setConnectionRequestTimeout(2000)// 设置从连接池获取连接的等待超时时间
        .build();
CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(pool)
                .evictIdleConnections(1, TimeUnit.MINUTES) //定期回收空闲连接
                .setDefaultRequestConfig(requestConfig) // 设置默认请求配置
                .evictExpiredConnections()//定期回收过期连接
                .build();

为什么要用Http连接池

1、降低延迟：如果不采用连接池，每次连接发起Http请求的时候都会重新建立TCP连接(经历3次握手)，用完就会关闭连接(4次挥手)，如果采用连接池则减少了这部分时间损耗。

2、支持更大的并发：如果不采用连接池，每次连接都会打开一个端口，在大并发的情况下系统的端口资源很快就会被用完，导致无法建立新的连接。
如短连接去做压测，可能会报错：java.net.NoRouteToHostException: Cannot assign requested address。

而我们一般用的长连接是通过添加Keep-Alive头参解决: header.add(new NVPair(“Connection”, “Keep-Alive”))；

而事实上头参中加了此参数只是保持长连接，与建立TCP连接保持的长连接不一样。

HttpClient connecttimeout与sockettimeout区别

HttpClient 作为 Http 工具包，是基于socket的封装，所以 HttpClient 的connectTimeout、socketTimeout其实都是socket的connectTimeout、socketTimeout。

connectTimeout： 建立链接的最大时间，也就是完成三次握手，建立TCP链接所用的时间。
socketTimeout： InputStream上调用read()方法阻塞的最大时间，即等待数据的时间或者两个包之间的间隔时间。

• 程序中最好设置connectTimeout、socketTimeout，可以防止阻塞
  • 如果不设置connectTimeout会导致，建立TCP链接时，阻塞，假死。
  • 如果不设置socketTimeout会导致，已经建立了TCP链接，在通信时，发送了请求报文，恰好此时，网络断掉，程序就阻塞，假死在那。
• 有时，connectTimeout并不像你想的那样一直到最大时间
  • socket建立链接时，如果网络层确定不可达，会直接抛出异常，不会一直到connectTimeout的设定值。
  • socket连接一个不存在的ip，很快就会提示连不上，而不是等到指定时间才报错。推测可能底层网络已经确定目标不可达所以不会一直重试直到connecttimeout。

HttpClient重试

关于HttpClient重试策略的研究

提供两种重试机制：

• HttpRequestRetryHandler：针对异常的重试（例如：connect timed out、read timed out）
• ServiceUnavailableRetryStrategy：针对响应错误码的重试（例如：400、500等）

HttpRequestRetryHandler

HttpClient提供针对异常重试的接口，实现该接口就可以编写异常重试方案。

DefaultHttpRequestRetryHandler

HttpClient提供的默认重试方案。

StandardHttpRequestRetryHandler

这是官方提供的一个标准的retry方案，为了保证幂等性约定Resetful接口必须是GET, HEAD, PUT, DELETE, OPTIONS, and TRACE中的一种。

自定义重试

实现 HttpRequestRetryHandler 接口，在 retryRequest 方法中编写重试逻辑。

public static CloseableHttpClient getHttpClient() {
    HttpRequestRetryHandler httpRequestRetryHandler = new HttpRequestRetryHandler() {
        // exception：异常，executionCount：重试次数
        // 返回true表示重试，返回false表示不需要重试
        public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
            System.out.println("重试第" + executionCount + "次");
            if (executionCount >= 3) {
                // Do not retry if over max retry count
                return false;
            }
            if (exception instanceof InterruptedIOException) {
                // Timeout
                return false;
            }
            if (exception instanceof UnknownHostException) {
                // Unknown host
                return false;
            }
            if (exception instanceof ConnectTimeoutException) {
                // Connection refused
                return false;
            }
            if (exception instanceof SSLException) {
                // SSL handshake exception
                return false;
            }
            HttpClientContext clientContext = HttpClientContext.adapt(context);
            HttpRequest request = clientContext.getRequest();
            boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
            if (idempotent) {
                // 如果请求被认为是幂等的，那么就重试
                return true;
            }
            return false;
        }
    };
    return HttpClients.custom().setRetryHandler(httpRequestRetryHandler)
            .setConnectionManager(new PoolingHttpClientConnectionManager()).build();
}

ServiceUnavailableRetryStrategy

针对响应错误码的重试，并且可以自定义重试时间的间隔。

public class MyServiceUnavailableRetryStrategy implements ServiceUnavailableRetryStrategy {
    private int executionCount;
    private long retryInterval;
    MyServiceUnavailableRetryStrategy(Builder builder) {
        this.executionCount = builder.executionCount;
        this.retryInterval = builder.retryInterval;
    }
    // retry逻辑
    @Override
    public boolean retryRequest(HttpResponse response, int executionCount, HttpContext context) {
        if (response.getStatusLine().getStatusCode() != 200 && executionCount < this.executionCount){
            // 表示错误码不是200并且小于最大重试次数进行重试
            return true;
        }
        else{
            return false;
        }
    }
    // retry间隔时间
    @Override
    public long getRetryInterval() {
        return this.retryInterval;
    }
    public static final class Builder {
        private int executionCount;
        private long retryInterval;
        public Builder() {
            executionCount = 3;
            retryInterval = 1000;
        }
        public Builder executionCount(int executionCount) {
            this.executionCount = executionCount;
            return this;
        }
        public Builder retryInterval(long retryInterval) {
            this.retryInterval = retryInterval;
            return this;
        }
        public MyServiceUnavailableRetryStrategy build() {
            return new MyServiceUnavailableRetryStrategy(this);
        }
    }
}

public static CloseableHttpClient getHttpClient() {
    ServiceUnavailableRetryStrategy serviceUnavailableRetryStrategy = new MyServiceUnavailableRetryStrategy.Builder()
            .executionCount(3)
            .retryInterval(1000)
            .build();
    return HttpClients.custom().setServiceUnavailableRetryStrategy(serviceUnavailableRetryStrategy)
            .setConnectionManager(new PoolingHttpClientConnectionManager()).build();
}

HttpClient封装

HttpClient用法–这一篇全了解（内含例子）
HttpClient详细使用示例

httpclient.properties 配置：

http.maxTotal=200
http.defaultMaxPerRoute=50
http.connectTimeout=2000
http.connectionRequestTimeout=500
http.socketTimeout=10000
http.staleConnectionCheckEnabled=true

Bean配置类：

import lombok.Getter;
import lombok.Setter;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;

import java.util.concurrent.TimeUnit;

@Configuration
@PropertySource(value = "classpath:httpclient.properties")
@Getter
@Setter
public class HttpClientConfig {
    @Value("${http.maxTotal}")
    private int maxTotal;
    @Value("${http.defaultMaxPerRoute}")
    private int defaultMaxPerRoute;
    @Value("${http.connectTimeout}")
    private int connectTimeout;
    @Value("${http.connectionRequestTimeout}")
    private int connectionRequestTimeout;
    @Value("${http.socketTimeout}")
    private int socketTimeout;
    @Value("${http.staleConnectionCheckEnabled}")
    private boolean staleConnectionCheckEnabled;

    @Bean
    public CloseableHttpClient httpClient(){
        PoolingHttpClientConnectionManager pool = new PoolingHttpClientConnectionManager();
        // 设置连接池的最大连接数
        pool.setMaxTotal(maxTotal);
        // 设置每个路由的最大连接数，路由是对MaxTotal的细分
        pool.setDefaultMaxPerRoute(defaultMaxPerRoute);
        return HttpClients.custom().setConnectionManager(pool)
                // 定期回收空闲连接
                .evictIdleConnections(1, TimeUnit.MINUTES)
                // 定期回收过期连接
                .evictExpiredConnections()
                .build();
    }

    @Bean
    public RequestConfig requestConfig(){
        return RequestConfig.custom()
                // 设置连接超时时间
                .setConnectTimeout(connectTimeout)
                // 设置等待数据超时时间
                .setSocketTimeout(socketTimeout)
                // 设置从连接池获取连接的等待超时时间
                .setConnectionRequestTimeout(connectionRequestTimeout)
                .build();
    }
}

Http响应结果类：

public class HttpResult {
    private Integer status;
    private String content;
    public HttpResult(){}

    public HttpResult(Integer status, String content) {
        this.status = status;
        this.content = content;
    }

    public Integer getStatus() {
        return status;
    }

    public void setStatus(Integer status) {
        this.status = status;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }
}

Http请求封装类：

import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpDelete;
import org.apache.http.client.methods.HttpEntityEnclosingRequestBase;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

@Component
public class HttpClientUtil {

    @Autowired(required = false)
    private CloseableHttpClient httpClient;

    @Autowired(required = false)
    private RequestConfig requestConfig;

    /**
     * 执行GET请求，如果请求状态不为200返回null
     *
     * @param url
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public String doGet(String url) throws ClientProtocolException, IOException {
        // 创建http GET请求
        HttpGet httpGet = new HttpGet(url);
        httpGet.setConfig(requestConfig);

        CloseableHttpResponse response = null;
        try {
            // 执行请求
            response = httpClient.execute(httpGet);
            // 判断返回状态是否为200
            if (response.getStatusLine().getStatusCode() == 200) {
                return EntityUtils.toString(response.getEntity(), "UTF-8");
            }
        } finally {
            if (response != null) {
                response.close();
            }
        }
        return null;
    }

    /**
     * 带参数的GET请求
     *
     * @param url
     * @param params
     * @return
     * @throws URISyntaxException
     * @throws ClientProtocolException
     * @throws IOException
     */
    public String doGet(String url, Map<String, Object> params) throws URISyntaxException,
            ClientProtocolException, IOException {
        // 拼接URL
        URIBuilder uriBuilder = new URIBuilder(url);
        if (null != params) {
            for (Map.Entry<String, Object> entry : params.entrySet()) {
                uriBuilder.setParameter(entry.getKey(), entry.getValue().toString());
            }
        }
        String newUrl = uriBuilder.build().toString();
        return doGet(newUrl);
    }

    /**
     * 执行doPOST请求，请求响应状态不为200返回null
     *
     * @param url
     * @param params
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doPost(String url, Map<String, Object> params) throws ClientProtocolException,
            IOException {
        // 创建http POST请求
        HttpPost httpPost = new HttpPost(url);
        httpPost.setConfig(requestConfig);
        setFormEntity(httpPost, params);
        return doHttp(httpPost);
    }

    /**
     * post提交json
     *
     * @param url
     * @param json
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doPostJson(String url, String json) throws ClientProtocolException, IOException {
        // 创建http POST请求
        HttpPost httpPost = new HttpPost(url);
        httpPost.setConfig(requestConfig);
        // 设置参数
        if (null != json) {
            // 构造一个form表单式的实体
            StringEntity stringEntity = new StringEntity(json, ContentType.APPLICATION_JSON);
            // 将请求实体设置到httpPost对象中
            httpPost.setEntity(stringEntity);
        }

        return doHttp(httpPost);
    }

    /**
     * 执行POST请求
     *
     * @param url
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doPost(String url) throws ClientProtocolException, IOException {
        return doPost(url, null);
    }

    /**
     * 执行doPut请求，请求响应状态不为200返回null
     *
     * @param url
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doPut(String url) throws ClientProtocolException, IOException {
        return doPut(url, null);
    }

    /**
     * 执行doPut请求，请求响应状态不为200返回null
     *
     * @param url
     * @param params
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doPut(String url, Map<String, Object> params) throws ClientProtocolException,
            IOException {
        // 创建http PUT请求
        HttpPut httpPut = new HttpPut(url);
        httpPut.setConfig(requestConfig);
        setFormEntity(httpPut, params);
        return doHttp(httpPut);
    }

    /**
     * 执行DELETE请求
     *
     * @param url
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doDelete(String url) throws ClientProtocolException, IOException {
        // 创建http PUT请求
        HttpDelete httpDelete = new HttpDelete(url);
        httpDelete.setConfig(requestConfig);
        return doHttp(httpDelete);
    }

    /**
     * 执行DELETE请求，使用_method指定DELETE请求，实际发送的POST请求
     *
     * @param url
     * @param params
     * @return
     * @throws ClientProtocolException
     * @throws IOException
     */
    public HttpResult doDelete(String url, Map<String, Object> params) throws ClientProtocolException,
            IOException {
        if (params == null) {
            params = new HashMap<String, Object>();
        }
        params.put("_method", "DELETE");
        return doPost(url, params);
    }

    private void setFormEntity(HttpEntityEnclosingRequestBase requestBase, Map<String, Object> params)
            throws UnsupportedEncodingException {
        // 设置参数
        if (null != params) {
            // 参数列表
            List<NameValuePair> parameters = new ArrayList<NameValuePair>(0);
            for (Map.Entry<String, Object> entry : params.entrySet()) {
                parameters.add(new BasicNameValuePair(entry.getKey(), entry.getValue().toString()));
            }
            // 构造一个form表单式的实体
            UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(parameters, "UTF-8");
            // 将请求实体设置到httpPost对象中
            requestBase.setEntity(formEntity);
        }
    }

    private HttpResult doHttp(HttpRequestBase requestBase) throws ClientProtocolException, IOException {
        CloseableHttpResponse response = null;
        try {
            // 执行请求
            response = httpClient.execute(requestBase);
            // 判断返回状态是否为200
            if (response.getStatusLine().getStatusCode() == 200) {
                return new HttpResult(200, EntityUtils.toString(response.getEntity(), "UTF-8"));
            } else {
                return new HttpResult(response.getStatusLine().getStatusCode(), null);
            }
        } finally {
            if (response != null) {
                response.close();
            }
        }
    }

}