JAVA如何正确使用爬虫代理

最新推荐文章于 2024-12-22 16:34:07 发布

原创最新推荐文章于 2024-12-22 16:34:07 发布 · 758 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#亿牛云范范

网络爬虫同时被 3 个专栏收录

206 篇文章

订阅专栏

数据采集

198 篇文章

订阅专栏

http代理

189 篇文章

订阅专栏

本文介绍了如何在Java中使用HttpClient3.1库配置代理进行爬虫操作。示例代码展示了如何设置代理服务器（PROXY_HOST: t.16yun.cn, PROXY_PORT: 31111）进行HTTP请求，并提供了相关类和方法的使用说明。如需了解更多详情，请访问www.16yun.cn。" 108882316,9721991,过拟合与欠拟合：原因、识别与解决方案,"['机器学习', '深度学习', '模型训练', '模型评估', '特征选择']

HttpClient3.1
import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;

import java.io.IOException;

public class Main {
private static final String PROXY_HOST = “t.16yun.cn”;
private static final int PROXY_PORT = 31111;

public static void main(String[] args) {
    HttpClient client = new HttpClient();
    HttpMethod method = new GetMethod("https://httpbin.org/ip");

    HostConfiguration config = client.getHostConfiguration();
    config.setProxy(PROXY_HOST, PROXY_PORT);

    client.getParams().setAuthenticationPreemptive(true);

    String username = "16ABCCKJ";
    String password = "712323";
    Credentials credentials = new UsernamePasswordCredentials(username, password);
    AuthScope authScope = new AuthScope(PROXY_HOST, PROXY_PORT);

    client.getState().setProxyCredentials(authScope, credentials);

    try {
        client.executeMethod(method);

        if (method.getStatusCode() == HttpStatus.SC_OK) {
            String response = method.getResponseBodyAsString();
            System.out.println("Response = " + response);
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        method.releaseConnection();
    }
}

}
点击此链接了解更多 www.16yun.cn