网络爬虫_网页登录（苏宁有验证码）—基于HtmlUnit

最新推荐文章于 2021-06-19 04:00:53 发布

原创最新推荐文章于 2021-06-19 04:00:53 发布 · 3.7k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#HtmlUnit #网络爬虫 #苏宁

网络爬虫学习笔记专栏收录该内容

3 篇文章

订阅专栏

本文介绍如何利用HtmlUnit库进行网络爬虫，针对有验证码的苏宁网页登录进行详细步骤演示，通过代码示例展示登录过程及数据抓取。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

输入验证码：

public static String getCode(){
    System.out.println("请输入验证码：");
    Scanner sc = new Scanner(System.in);
    String code = sc.nextLine();
    return code;
}

初始化WebClient方法：

public WebClient getClient(){
    webClient = new WebClient(BrowserVersion.FIREFOX_24);
    webClient.getOptions().setCssEnabled(false);
    webClient.getOptions().setJavaScriptEnabled(true);
    webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    webClient.setAjaxController(new NicelyResynchronizingAjaxController());
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    return webClient;
}

爬取数据代码示例：

/**
 * Create by lingfx on 2017/8/18
 */
public class SuNingLoginTest {

    private Logger logger = Logger.getLogger(SuNingLoginTest.class);
    private WebClient client;
    private String username = "用户名";
    private String kkk = "密码";

    private String filePath = "E://code.jpg";
    public void login() throws IOException {        client = new WebClientTool().getClient();        HtmlPage htmlPage = client.getPage("https://mpassport.suning.com/ids/login?service=https%3A%2F%2Fscs.suning.com%2Fsps%2Fauth%3FtargetUrl%3Dhttp%253A%252F%252Fscs.suning.com%252Fsps%252Fmember%252Flogon.do&loginTheme=scs");//        logger.info(htmlPage.asText());        HtmlInput un = htmlPage.getHtmlElementById("username");        un.setValueAttribute(username);        HtmlInput pwd = htmlPage.getHtmlElementById("pwd");        pwd.setValueAttribute(kkk);        HtmlImage codeImg = htmlPage.getHtmlElementById("vcodeimg1");
	codeImg.saveAs(new File(filePath));
	logger.info("验证码存储位置："+filePath);
        String code = ScannerTool.getCode();        HtmlInput codeInput = htmlPage.getHtmlElementById("validateCode");        codeInput.setValueAttribute(code);                ScriptResult res  = htmlPage.executeJavaScript("javascript:logon();");        HtmlPage page2 = (HtmlPage) res.getNewPage();        logger.info(page2.asText());        if(page2.asText().contains("您好！欢迎回来！")){            logger.info("登录成功！");        }    }        public static void main(String[] args) throws IOException {        SuNingLoginTest test = new SuNingLoginTest();        test.login();    }}