spark入门

Using a Mesos Master URL

The Master URLs for Mesos are in the form mesos://host:5050 for a single-master Mesos cluster, ormesos://zk://host:2181 for a multi-master Mesos cluster using ZooKeeper.

The driver also needs some configuration in spark-env.sh to interact properly with Mesos:

1. In spark.env.sh set some environment variables:
   • export MESOS_NATIVE_LIBRARY=<path to libmesos.so>. This path is typically <prefix>/lib/libmesos.so where the prefix is /usr/local by default. See Mesos installation instructions above. On Mac OS X, the library is called libmesos.dylib instead of libmesos.so.
   • export SPARK_EXECUTOR_URI=<URL of spark-1.0.1.tar.gz uploaded above>.
2. Also set spark.executor.uri to <URL of spark-1.0.1.tar.gz>.

edit /etc/profle append :

export MESOS_NATIVE_LIBRARY="/usr/local/lib/libmesos.so"

 

Now when starting a Spark application against the cluster, pass a mesos:// URL as the master when creating aSparkContext. For example:

val conf = new SparkConf()
  .setMaster("mesos://HOST:5050")
  .setAppName("My app")
  .set("spark.executor.uri", "<path to spark-1.0.1.tar.gz uploaded above>")
val sc = new SparkContext(conf)

(You can also use spark-submit and configure spark.executor.uri in the conf/spark-defaults.conf file. Note that spark-submit currently only supports deploying the Spark driver in client mode for Mesos.)

When running a shell, the spark.executor.uri parameter is inherited from SPARK_EXECUTOR_URI, so it does not need to be redundantly passed in as a system property.

./bin/spark-shell --master mesos://host:5050

Mesos Run Modes

Spark can run over Mesos in two modes: “fine-grained” (default) and “coarse-grained”.

In “fine-grained” mode (default), each Spark task runs as a separate Mesos task. This allows multiple instances of Spark (and other frameworks) to share machines at a very fine granularity, where each application gets more or fewer machines as it ramps up and down, but it comes with an additional overhead in launching each task. This mode may be inappropriate for low-latency requirements like interactive queries or serving web requests.

The “coarse-grained” mode will instead launch only one long-running Spark task on each Mesos machine, and dynamically schedule its own “mini-tasks” within it. The benefit is much lower startup overhead, but at the cost of reserving the Mesos resources for the complete duration of the application.

To run in coarse-grained mode, set the spark.mesos.coarse property in your SparkConf:

conf.set("spark.mesos.coarse", "true")

In addition, for coarse-grained mode, you can control the maximum number of resources Spark will acquire. By default, it will acquire all cores in the cluster (that get offered by Mesos), which only makes sense if you run just one application at a time. You can cap the maximum number of cores using conf.set("spark.cores.max", "10") (for example).

 

启动spark：

./bin/spark-shell --master mesos://127.0.1.1:5050

#测试

scala> val file = sc.textFile("hdfs://hadoop-master:9000/tmp/WifiScan_None_20140723.csv")

scala> val count=file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)

scala> count.count()

http://spark.apache.org/docs/latest/running-on-mesos.html