DevOps(6)Spark Deployment on VM 2

DevOps(6)Spark Deployment on VM 2

 

1. Workers Sync from Master

under SPARK_HOME/conf/spark-env.sh

 

It works in cluster mode.

#SCALA_HOME=/opt/scala/scala-2.10.3

export SPARK_WORKER_MEMORY=768m

#SPARK_CLASSPATH='/opt/profiles-spark/lib/*.jar'

export SPARK_JAVA_OPTS="-Dbuild.env=sparkvm"

#export SPARK_DAEMON_JAVA_OPTS="-Dbuild.env=sparkvm"

#jobJar='/opt/profiles-spark/lib/*.jar'

#export JAVA_OPTS="-Dbuild.env=sparkvm"

export USER=carl

 

It requires we extend the serialization interface.

 

2. Pre-place the binary files

Did not try that. I do not like this way.

 

 

References:

http://sillycat.iteye.com/blog/2166583