《快学Scala》, <Scala For the Impatient>习题第三章

本文介绍Scala中数组操作的实用技巧,包括生成随机整数数组、交换数组元素、按条件重新排序数组元素及计算Double数组平均值的方法。
1. Write a code snippet that sets a to an array of n random integers between 0 (inclusive) and n (exclusive).
  def genRandomArray(n: Int) : Array[Int] = for (i <- 0.until(n).toArray) yield scala.util.Random.nextInt(n - 1)
2. Write a loop that swaps adjacent elements of an array of integers. For example, Array(1, 2, 3, 4, 5) becomes Array(2, 1, 4, 3, 5).

scala> :paste
// Entering paste mode (ctrl-D to finish)

 def swapArrayInPlace(arr: Array[Int]): Unit = {
    var temp = 0;
    for (i <- 0.until(arr.length, 2)) {
      if (i < arr.length - 2) {
        temp = arr(i + 1)
        arr(i + 1) = arr(i)
        arr(i) = temp
      }
    }
  }


// Exiting paste mode, now interpreting.

swapArrayInPlace: (arr: Array[Int])Unit
scala> val a = Array(1,2,3,4,5)
a: Array[Int] = Array(1, 2, 3, 4, 5)

scala> swapArrayInPlace(a)

scala> a
res71: Array[Int] = Array(2, 1, 4, 3, 5)
3. Repeat the preceding assignment, but produce a new array with swapped values. Use for/yield.
scala> :paste
// Entering paste mode (ctrl-D to finish)

  def swapArray(arr: Array[Int]) : Array[Int] = for (i <- arr.indices.toArray) yield {
    if (i == arr.length - 1)
      arr(i)
    else if (i % 2 == 0)
      arr(i+1)
    else arr(i-1)
  }

// Exiting paste mode, now interpreting.

swapArray: (arr: Array[Int])Array[Int]

scala> swapArray(Array(1,2,3,4,5))
res60: Array[Int] = Array(2, 1, 4, 3, 5)
4. Given an array of integers, produce a new array that contains all positive values of the original array, in their original order, followed by all values that are zero or negative, in their original order.
  def reorder(arr: Array[Int]) : Array[Int] = {
    arr.filter(_ > 0) ++ arr.filter(_ <= 0)
  }

scala> reorder(Array(4,-1,0, 52,15,6,-7))
res72: Array[Int] = Array(4, 52, 15, 6, -1, 0, -7)
5. How do you compute the average of an Array[Double]?
<?xml version="1.0" encoding="UTF-8"?> 4.0.0 <groupId>org.example</groupId> <artifactId>scala_tutorial</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <scala.version>2.12.11</scala.version> <spark.version>3.1.1</spark.version> <hadoop.version>3.2.4</hadoop.version> </properties> <dependencies> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>8.0.33</version> </dependency> <!-- Scala Library --> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <!-- Spark Core --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>${spark.version}</version> </dependency> <!-- Spark SQL --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>${spark.version}</version> </dependency> <!-- Spark Hive Support --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.12</artifactId> <version>${spark.version}</version> </dependency> <!-- Hadoop Client --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> </dependency> </dependencies> <build> <plugins> <!-- Scala Compiler Plugin --> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>4.8.1</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> <configuration> <scalaVersion>${scala.version}</scalaVersion> </configuration> </plugin> <!-- Maven Compiler Plugin --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <source>8</source> <target>8</target> </configuration> </plugin> <!-- Maven Assembly Plugin for creating fat JAR --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>3.6.0</version> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> 版本不变,给我调整个能用的
最新发布
11-22
你提供的 `pom.xml` 整体结构是正确的,但存在几个 **关键问题** 会导致: - 日志冲突(SLF4J 多绑定) - Hadoop/Spark 兼容性问题 - 打包后运行出错(缺少主类定义、依赖冲突) --- ### ✅ 目标 在 **不改变版本** 的前提下(Scala 2.12.11, Spark 3.1.1, Hadoop 3.2.4),调整 `pom.xml` 使其: 1. 能正常编译 Scala 代码 2. 消除 SLF4J 冲突警告 3. 打包成可执行的 fat jar(包含依赖) 4. 支持本地运行 Spark + Hive 5. 避免常见类加载冲突 --- ### ✅ 修改后的完整 `pom.xml` ```xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.example</groupId> <artifactId>scala_tutorial</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <scala.version>2.12.11</scala.version> <spark.version>3.1.1</spark.version> <hadoop.version>3.2.4</hadoop.version> <main.class>MySQLToHiveIncremental2</main.class> <!-- 主类名 --> </properties> <dependencies> <!-- MySQL Connector --> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>8.0.33</version> </dependency> <!-- Scala Library --> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <!-- Spark Core --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>${spark.version}</version> <exclusions> <!-- 排除日志实现冲突 --> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <!-- 避免与 Hadoop 中的 guava 版本冲突 --> <exclusion> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> </exclusion> </exclusions> </dependency> <!-- Spark SQL --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>${spark.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-reload4j</artifactId> </exclusion> </exclusions> </dependency> <!-- Spark Hive Support --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.12</artifactId> <version>${spark.version}</version> </dependency> <!-- Hadoop Client --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> <exclusions> <!-- 统一日志框架 --> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-reload4j</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <!-- 使用项目中统一的 Guava --> <exclusion> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> </exclusion> </exclusions> </dependency> <!-- 统一使用 log4j-over-slf4j + reload4j 实现 --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.30</version> </dependency> <dependency> <groupId>ch.qos.reload4j</groupId> <artifactId>reload4j</artifactId> <version>1.2.18.3</version> </dependency> <!-- 将 log4j API 桥接到 slf4j --> <dependency> <groupId>org.slf4j</groupId> <artifactId>log4j-over-slf4j</artifactId> <version>1.7.30</version> </dependency> <!-- Google Guava(避免多个版本冲突) --> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>27.0-jre</version> </dependency> </dependencies> <build> <sourceDirectory>src/main/scala</sourceDirectory> <testSourceDirectory>src/test/scala</testSourceDirectory> <plugins> <!-- Scala Compiler Plugin --> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>4.8.1</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> <configuration> <scalaVersion>${scala.version}</scalaVersion> </configuration> </plugin> <!-- Maven Compiler Plugin --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <source>8</source> <target>8</target> </configuration> </plugin> <!-- Maven Assembly Plugin: 构建可执行 fat jar --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>3.6.0</version> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <mainClass>${main.class}</mainClass> <!-- 设置主类 --> </manifest> </archive> <finalName>scala_tutorial</finalName> <appendAssemblyId>false</appendAssemblyId> <!-- 不加 -jar-with-dependencies --> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> ``` --- ### ✅ 关键修改点解释 | 修改项 | 说明 | |------|------| | `<packaging>jar</packaging>` | 明确打包类型 | | `<sourceDirectory>` | 告诉 Maven 正确扫描 Scala 源码目录 | | 排除 `slf4j-log4j12`, `slf4j-reload4j`, `log4j` 等冲突依赖 | 防止“Multiple SLF4J bindings”错误 | | 添加 `log4j-over-slf4j` + `reload4j` | 统一日志输出机制 | | 统一引入 `guava` 并排除其他模块中的版本 | 防止 `NoSuchMethodError` | | 在 `maven-assembly-plugin` 中设置 `<mainClass>` | 生成的 JAR 可直接用 `java -jar xxx.jar` 运行 | | `<appendAssemblyId>false</appendAssemblyId>` | 输出文件名为 `scala_tutorial.jar` 而不是 `xxx-jar-with-dependencies.jar` | --- ### ✅ 编译和运行方式 #### 1. 编译打包: ```bash mvn clean package ``` ✅ 输出:`target/scala_tutorial.jar` #### 2. 运行(IDEA 或命令行): ```bash java -cp target/scala_tutorial.jar MySQLToHiveIncremental2 ``` 或直接运行: ```bash spark-submit --class MySQLToHiveIncremental2 target/scala_tutorial.jar ``` --- ### ✅ 补充建议 1. **添加 `log4j.properties` 文件** - 在 `src/main/resources/` 下新建文件:`log4j.properties` - 内容见上一个回答,否则仍有 `No appenders could be found` 警告 2. **确保你的主类中有 `.master("local[*]")`** ```scala val spark = SparkSession.builder() .appName("MySQLToHive") .master("local[*]") // 必须有! .enableHiveSupport() .getOrCreate() ``` --- ### ✅ 成功标志 当你看到以下日志时,表示成功启动: ``` INFO SparkContext: Running Spark version 3.1.1 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://xxx:4040 ``` ---
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值