java.net.HttpURLConnection

Java网页抓取与Cookie管理

最新推荐文章于 2024-01-12 11:36:56 发布

最新推荐文章于 2024-01-12 11:36:56 发布 · 179 阅读

文章标签：

#.net #Java #OS #IE #Windows

本文介绍如何使用Java进行网页内容抓取及Cookie管理，包括通过URL建立连接、读取网页内容、获取和设置Cookie等操作。还介绍了POST请求的模拟发送。

1.网页内容获取
java.io.inputstream in;
java.net.url url = new java.net.url(www.xyz.com/content.html);
java.net.httpurlconnection connection = (java.net.httpurlconnection)
url.openconnection();
connection = (java.net.httpurlconnection) url.openconnection();
//模拟成ie
connection.setrequestproperty("user-agent","mozilla/4.0 (compatible; msie 6.0; windows 2000)");
connection.connect();
in = connection.getinputstream();
java.io.bufferedreader breader =
new bufferedreader(new inputstreamreader(in , "gbk"));
string str=breader.readline());
while(st != null){
system.out.println(str);
str=breader.readline());
}
2.cookie管理

1.直接的方式
取得cookie：
httpurlconnection huc= (httpurlconnection) url.openconnection();
inputstream is = huc.getinputstream();
// 取得sessionid.
string cookieval = hc.getheaderfield("set-cookie");
string sessionid;
if(cookieval != null)
{
sessionid = cookieval.substring(0, cookieval.indexof(";"));
}

发送设置cookie：
httpurlconnection huc= (httpurlconnection) url.openconnection();
if(sessionid != null)
{
huc.setrequestproperty("cookie", sessionid);
}
inputstream is = huc.getinputstream();

2.利用的jcookie包(http://jcookie.sourceforge.net/ )
获取cookie:
url url = new url("http://www.site.com/");
httpurlconnection huc = (httpurlconnection) url.openconnection();
huc.connect();
inputstream is = huc.getinputstream();
client client = new client();
cookiejar cj = client.getcookies(huc);

新的请求,利用上面获取的cookie:

url = new url("http://www.site.com/");
huc = (httpurlconnection) url.openconnection();
client.setcookies(huc, cj);

3.post方式的模拟
url url = new url("www.xyz.com");
httpurlconnection huc = (httpurlconnection) url.openconnection();
//设置允许output
huc.setdooutput(true);
//设置为post方式
huc.setrequestmethod("post");
huc.setrequestproperty("user-agent","mozilla/4.7 [en] (win98; i)");
stringbuffer sb = new stringbuffer();
sb.append("username="+usernme);
sb.append("&password="+password);

//post信息
outputstream os = huc.getoutputstream();
os.write(sb.tostring().getbytes("gbk"));
os.close();

bufferedreader br = new bufferedreader(new inputstreamreader(huc.getinputstream()))

huc.connect();

string line = br.readline();

while(line != null){

system.out.printli(line);

line = br.readline();

}