页面下载器(我的Java爬虫之一)

最新推荐文章于 2025-05-01 14:12:30 发布

转载最新推荐文章于 2025-05-01 14:12:30 发布 · 111 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://segmentfault.com/a/1190000011881759

文章标签：

#爬虫 #java

本文详细介绍Maven打包的各种方式，包括官方推荐的打包方法、自定义打包配置及如何创建可执行的Jar包。此外还介绍了使用Maven进行网页下载器开发的方法。

说点别的

maven打包

官方定制的打包方式

使用maven assembly plugin插件完成打包操作，插件配置在pom.xml文件的build标签中，格式如下。

<build>
    [...]
    <plugins>
      <plugin>
        <!-- NOTE: We don't need a groupId specification because the group is
             org.apache.maven.plugins ...which is assumed by default.
         -->
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>

executions用于将目标和maven的某个生命周期进行绑定

<executions>
  <execution>
    <id>make-assembly</id> <!-- this is used for inheritance merges -->
    <phase>package</phase> <!-- bind to the packaging phase -->
    <goals>
      <goal>single</goal>
    </goals>
  </execution>
</executions>

创建可执行的jar包

<build>
   [...]
   <plugins>
     <plugin>
       <artifactId>maven-assembly-plugin</artifactId>
       <version>3.1.0</version>
       <configuration>
         [...]
         <archive>
           <manifest>
             <mainClass>org.sample.App</mainClass>
           </manifest>
         </archive>
       </configuration>
       [...]
     </plugin>
     [...]

自定义打包方式

上文已提到使用官方定制的打包方式，使用<descriptorRefs></descriptorRefs>标签即可；如果使用自定义的打包方式，使用<descriptors></descriptors>标签。

<project>
  [...]
  <build>
    [...]
    <plugins>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <descriptors>
            <descriptor>src/assembly/src.xml</descriptor>
          </descriptors>
        </configuration>
        [...]
</project>

src.xml的格式大致如下

<assembly
        xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
    <id>snapshot</id>
    <formats>
        <format>jar</format>
    </formats>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/lib</outputDirectory>
        </dependencySet>
    </dependencySets>
</assembly>

使用<fileSets>允许用户通过文件或目录的粒度来控制打包，往往配置一个bin目录，里面存放可运行的脚本。这种方法打成的包如何运行？
两种方法：

将依赖通过cp全部指定，然后运行，java -cp lib/dependency1:lib/dependency2 类全名
java -Djava.ext.dirs=lib 类命名，此方法貌似java 9不再支持

页面下载器

前期准备

maven导入依赖

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.3</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>fluent-hc</artifactId>
    <version>4.5.3</version>
</dependency>

下载器第一版

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.nio.charset.Charset;

public void testGet1() {
  CloseableHttpClient clients = HttpClients.createDefault();
  RequestBuilder builder = requestBuilder.get("http://www.qq.com");
  HttpGet httpGet = new HttpGet(builder.build().getURI());
  CloseableHttpResponse execute = null;
  try {
    execute = clients.execute(httpGet);
    HttpEntity entity = execute.getEntity();
    //此处可以自己写个charset的解析方法
    String page = EntityUtils.toString(entity);
    System.out.println(page);
  } catch (Exception e) {
    e.printStackTrace();
  } finally {
    if (execute != null) {
      try {
        execute.close();
      } catch (IOException e) {
        e.printStackTrace();
      }
    }
  }
}

第二版

匿名内部类版本

public void testGet2() {
  CloseableHttpClient clients = HttpClients.createDefault();
  RequestBuilder builder = RequestBuilder.get("http://www.qq.com");
  HttpGet httpGet = new HttpGet(builder.build().getURI());
  try {
    String page = clients.execute(httpGet, new ResponseHandler<String>() {
      @Override
      public String handleResponse(HttpResponse HttpResponse) throws ClientProtocolException, IOException
      HttpEntity entity = httpResponse.getEntity();
      String s = EntityUtils.toString(entity);
      return s;
    });
    System.out.println(page);
  } catch (Exception e) {
    e.printStackTrace();
  }
}

匿名内部类可以使用lambda表达式来替代，写法为

String page = clients.execute(httpGet, (HttpResponse HttpResponse) -> {
    HttpEntity entity = HttpResponse.getEntity();
    String s = EntityUtils.toString(entity);
    return s;
  });

第三版

使用org.apache.http.client.fluent包中的api

public void testGet3() {
  Response response = Request.Get("http://www.qq.com").execute();
  String s = response.returnContent().asString(Charset.forName("gb2312"));
  System.out.println(s);
}