spark0.9.0 安装 ubuntu12
11 March 2014
author:xiajun
spark0.9.0依赖 scala2.10.3
1.安装scala
下载scala2.10.3 http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
将其解压到 /usr/local 下,然后配置环境变量
vim /etc/profile
export SCALA_HOME=/usr/local/scala-2.10.3
export PATH=$PATH:$SCALA_HOME/bin
2.安装spark-0.9.0
下载spark-0.9.0 http://d3kbcqa49mib13.cloudfront.net/spark-0.9.0-incubating-bin-hadoop2.tgz
解压spark压缩包
tar -zvxf spark-0.9.0-incubating-bin-hadoop2.tgz
cd spark-0.9.0-incubating-bin-hadoop2/conf
修改:spark-env.sh
vim spark-env.sh
export JAVA_HOME=/usr/local/jd/jdk1.6.0_25 //jdk 安装目录
export SCALA_HOME=/usr/local/jd/scala-2.10.3 //scala 安装目录
export SPARK_MASTER_IP=10.12.117.196 //spark master 的ip
修改:slaves 将每个hadoop datanode 一行一个的写入此文件
启动:
./sbin/start-all.sh
yarn中运行:
SPARK_JAR=/usr/local/jd/spark-0.9.0/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
./bin/spark-class org.apache.spark.deploy.yarn.Client
--jar /usr/local/jd/spark-0.9.0/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar
--class org.apache.spark.examples.SparkPi
--args yarn-standalone
--num-workers 3
--master-memory 1g
--worker-memory 1g
--worker-cores 1
blog comments powered by Disqus