跟我一步一步学爬虫---传参篇（二）

今天我们来学习下用post方式请求后台。

我设计的程序有两种：
1、给服务器传参，然后服务器打印出来，然后用PrintWriter类写出来（模拟ajax）。

2、请求服务器，然后跳转到其他网页。

先做第1种（后台服务器用Servlet）：

java代码

import java.io.IOException;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.PostMethod;

public class PostMethodTest {
	
	public static void main(String[] args) {
		Test();
	}
	
	public static void Test() {
			
			HttpClient httpClient = new HttpClient();
			PostMethod postMethod = new PostMethod("http://localhost:8888/LODOP/b");
			NameValuePair pair[] = new NameValuePair[2];
			pair[0] = new NameValuePair("a", "123");
			pair[1] = new NameValuePair("b", "456");
			postMethod.addParameters(pair);
			try {
				int statusCode = httpClient.executeMethod(postMethod);
				System.out.println(statusCode);
				System.out.println(postMethod.getResponseBodyAsString());
			} catch (HttpException e) {
				e.printStackTrace();
			} catch (IOException e) {
				e.printStackTrace();
			}
	}
}

服务器代码：

package com.servlet;

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class aa extends HttpServlet {
	@Override
	protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
		String a = request.getParameter("a");
		String b = request.getParameter("b");
		System.out.println(a);
		System.out.println(b);
		PrintWriter pw = response.getWriter();
		pw.print("234,12,");
		pw.flush();
		pw.close();
	}
	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {
		// TODO Auto-generated method stub
		doPost(req, resp);
	}
}

执行main方法，服务器那边会打印123和456 main方法这边会打印状态码200和返回的234,12,

然后做第二种：

java代码

import java.io.IOException;

import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.methods.PostMethod;

public class PostMethodTest {
	
	public static void main(String[] args) {
		Test();
	}
	
	public static void Test() {
			
			HttpClient httpClient = new HttpClient();
			PostMethod postMethod = new PostMethod("http://localhost:8888/LODOP/b");
			try {
				int statusCode = httpClient.executeMethod(postMethod);
				System.out.println(statusCode);
				Header header = postMethod.getResponseHeader("location");
				System.out.println(header.getValue());
			} catch (HttpException e) {
				e.printStackTrace();
			} catch (IOException e) {
				e.printStackTrace();
			}
	}
}

服务器代码

package com.servlet;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class aa extends HttpServlet {
	@Override
	protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
		response.sendRedirect("http://www.baidu.com");
	}
	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {
		// TODO Auto-generated method stub
		doPost(req, resp);
	}
}

执行完main方法后会返回状态码203 表示重定向了

这个是时候我们需要从Header里面提取重定向的地址，然后重新抓取页面。

web.xml配置

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.4" 
	xmlns="http://java.sun.com/xml/ns/j2ee" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee 
	http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
	<welcome-file-list>
		<welcome-file>index.jsp</welcome-file>
	</welcome-file-list>
	
	<servlet>
		<servlet-name>b</servlet-name>
		<servlet-class>com.servlet.aa</servlet-class>
	</servlet>
	<servlet-mapping>
		<servlet-name>b</servlet-name>
		<url-pattern>/b</url-pattern>
	</servlet-mapping>
</web-app>