spark idea 的配置问题

最新推荐文章于 2025-05-06 15:03:26 发布

shiter

最新推荐文章于 2025-05-06 15:03:26 发布

阅读量8.8k

点赞数

CC 4.0 BY-SA版权

文章标签： spark

本文链接：https://blog.youkuaiyun.com/wangyaninglm/article/details/51030711

本文探讨了在使用Apache Spark过程中遇到的环境配置问题及错误信息解析，包括缺失或无效依赖的问题，以及如何正确地配置Spark环境来避免此类错误。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

不知道下面的错误是为什么？

Error:scalac: missing or invalid dependency detected while loading class file 'RDD.class'.
Could not access term hadoop in package org.apache,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'RDD.class' was compiled against an incompatible version of org.apache.

我i下面这个version应该写啥？写1.6.1 不对啊，没有找到

想跑一下下面的代码，死活有问题，哎，我就很讨厌java这套东西，环境配置半天不说，还慢的要死

/**
  * Created by Administrator on 2016/3/31.
  */

import org.apache.spark.{SparkConf, SparkContext}
/**
  * Created by Administrator on 2016/3/31.
  */
import scala.math.random

import org.apache.spark._
object simpleApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Spark Pi").setMaster("local")
    val spark = new SparkContext(conf)
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    val count = spark.parallelize(1 until n, slices).map { i =>
        val x = random * 2 - 1
        val y = random * 2 - 1
        if (x*x + y*y < 1) 1 else 0
      }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
    spark.stop()
  }
}

下面分享一个ibm 大数据基础的培训资料：

http://download.youkuaiyun.com/detail/wangyaninglm/9478412