11 March 2014

author:xiajun

spark0.9.0依赖 scala2.10.3

1.安装scala

下载scala2.10.3 http://www.scala-lang.org/files/archive/scala-2.10.3.tgz

将其解压到 /usr/local 下,然后配置环境变量

vim /etc/profile
export SCALA_HOME=/usr/local/scala-2.10.3
export PATH=$PATH:$SCALA_HOME/bin

2.安装spark-0.9.0

下载spark-0.9.0 http://d3kbcqa49mib13.cloudfront.net/spark-0.9.0-incubating-bin-hadoop2.tgz

解压spark压缩包

tar -zvxf spark-0.9.0-incubating-bin-hadoop2.tgz
cd spark-0.9.0-incubating-bin-hadoop2/conf
修改:spark-env.sh
vim spark-env.sh
export JAVA_HOME=/usr/local/jd/jdk1.6.0_25 //jdk 安装目录
export SCALA_HOME=/usr/local/jd/scala-2.10.3 //scala 安装目录
export SPARK_MASTER_IP=10.12.117.196 //spark master 的ip
修改:slaves 将每个hadoop datanode 一行一个的写入此文件

启动:

./sbin/start-all.sh 

yarn中运行:

 SPARK_JAR=/usr/local/jd/spark-0.9.0/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar  
./bin/spark-class org.apache.spark.deploy.yarn.Client 
--jar /usr/local/jd/spark-0.9.0/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar   
--class org.apache.spark.examples.SparkPi 
--args yarn-standalone 
--num-workers 3 
--master-memory 1g 
--worker-memory 1g 
--worker-cores 1


blog comments powered by Disqus