java 读取站点页面源码方法总结

最新推荐文章于 2021-03-05 00:35:33 发布

最新推荐文章于 2021-03-05 00:35:33 发布 · 99 阅读

文章标签：

#java

本文介绍了使用Java进行网页抓取的三种实用方法：通过Socket直接访问、利用HttpClient库以及使用HttpURLConnection。每种方法都提供了详细的代码示例，并解释了其应用场景。

java 读取页面源码的多种方式

以下都是实战经验：
1、Socket读取
String strServer= http://www.google.cn;//这里同样可以用ip来访问:203.208.35.100
String strPage="/language_tools?hl=zh-CN";
try {
String hostname = strServer;
int port = 80;
InetAddress addr = InetAddress.getByName(hostname);
Socket socket = new Socket(addr, port);
BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
wr.write("GET " + strPage + " HTTP/1.0/r/n");
wr.write("HOST:" + strServer + "/r/n");
wr.write("/r/n");
wr.flush();
BufferedReader rd = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
wr.close();
rd.close();
} catch (Exception e) {
System.out.println(e.toString());
}
2、HttpClient方式
HttpClient client=new HttpClient();
GetMethod method=new GetMethod(" http://www.baidu.com/");
int status=client.executeMethod(method);
if(status==HttpStatus.SC_OK){
//读取内容
byte[] responseBody = method.getResponseBody();
//处理内容
System.out.println(new String(responseBody));
System.out.println("文件名称："+method.getPath());
}
3、HttpURLConnection方式
URL url = new URL("这里是你要连接的地址");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setDoOutput(true);//是否可用于输出(输出参数)，默认为fasle。另:setDoInput()为是否可用于输入，默认为true
String parameters = "name=admin&password=123456";//这里是要传递的参数
OutputStream os = conn.getOutputStream();
os.write(parameters.getBytes("utf-8"));
os.flush();
os.close();
System.out.println("返回状态码："+conn.getResponseCode());
System.out.println("返回消息："+conn.getResponseMessage());
InputStream is = conn.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is,"utf-8"));
String line = null;
while((line=br.readLine())!=null){
System.out.println(line);
}
//DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();;
//DocumentBuilder db = dbf.newDocumentBuilder();
//Document doc = db.parse(is);
如果谁还有更多的方式分享，请留言！

注：HttpClient和GetMethod都是来自apache的包