【httpclient编写爬虫】post提交json数据和普通键值

写在开头

在开发爬虫的过程中，难免碰到post提交的问题。
本文比较了两种数据提交方式，并且使用httpclient模拟网站post提交两种数据。

我见过的post提交方式有两种：

1. 普通的键值对提交方式；
2. 提交json数据。

我所使用的httpclient版本

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
</dependency>

普通键值对的提交方式

CloseableHttpClient httpclient = HttpClients.createDefault();

HttpPost httpPost = new HttpPost("http://targethost/login");
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add(new BasicNameValuePair("username", "vip"));
nvps.add(new BasicNameValuePair("password", "secret"));
httpPost.setEntity(new UrlEncodedFormEntity(nvps));
CloseableHttpResponse response2 = httpclient.execute(httpPost);

try {
    System.out.println(response2.getStatusLine());
    HttpEntity entity2 = response2.getEntity();
    // do something useful with the response body
    // and ensure it is fully consumed
    EntityUtils.consume(entity2);
} finally {
    response2.close();
}

JSON数据提交方式

要提交的数据

{
    "username" : "vip",
    "password" : "secret"
}

代码

import org.apache.http.Consts;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.IOException;

/**
 * Created by CarlZhang on 2017/1/1.
 */
public class PostJsonTest {
    public static void main(String[] args) {
        CloseableHttpClient httpclient = HttpClients.createDefault();
        try {

            HttpPost httpPost = new HttpPost("http://targethost/login");

            //json数据{"username":"vip","password":"secret"}
            String jsonStr  = "{\"username\":\"vip\",\"password\":\"secret\"}";

            StringEntity se = new StringEntity(jsonStr, Consts.UTF_8);
            se.setContentEncoding("UTF-8");
            se.setContentType("application/json");

            httpPost.setEntity(se);
            CloseableHttpResponse response2 = httpclient.execute(httpPost);

            try {
                System.out.println(response2.getStatusLine());
                HttpEntity entity2 = response2.getEntity();
                // do something useful with the response body
                // and ensure it is fully consumed
                //EntityUtils.consume(entity2);
                String res = EntityUtils.toString(entity2);
                System.out.println(res);
            } finally {
                response2.close();
            }

        } catch (IOException e) {
            e.printStackTrace();
        }finally {
            try {
                httpclient.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

实例-JSON提交

如下是我在某个网站点击发帖，然后在chrome （按住F12键）打开的dubug工具，可以看到我提交的post请求。其中Form Data就是使用post提交的json数据。

后端怎么拿到这些数据的呢？
该网站使用的开源库latke中的Requests类
可以看到，它是通过Reader对象去读入流数据的。

 /**
     * Gets the request json object with the specified request.
     *
     * @param request the specified request
     * @param response the specified response, sets its content type with "application/json"
     * @return a json object
     * @throws ServletException servlet exception
     * @throws IOException io exception
     */
    public static JSONObject parseRequestJSONObject(final HttpServletRequest request, final HttpServletResponse response)
        throws ServletException, IOException {
        response.setContentType("application/json");

        final StringBuilder sb = new StringBuilder();
        BufferedReader reader;

        final String errMsg = "Can not parse request[requestURI=" + request.getRequestURI() + ", method=" + request.getMethod()
            + "], returns an empty json object";

        try {
            try {
                reader = request.getReader();
            } catch (final IllegalStateException illegalStateException) {
                reader = new BufferedReader(new InputStreamReader(request.getInputStream()));
            }

            String line = reader.readLine();

            while (null != line) {
                sb.append(line);
                line = reader.readLine();
            }
            reader.close();

            String tmp = sb.toString();

            if (Strings.isEmptyOrNull(tmp)) {
                tmp = "{}";
            }

            return new JSONObject(tmp);
        } catch (final Exception ex) {
            LOGGER.log(Level.ERROR, errMsg, ex);

            return new JSONObject();
        }
    }

另外，前端js代码是通过jquery的类库去提交的json数据

实例-普通post提交

有兴趣可以查看我这篇文章【js类库AngularJs】解决angular+springmvc的post提交问题