Ubuntu下Spark开发环境搭建

字数 3162阅读 1202评论 0赞 0

<ul><li>Ubuntu 64基本环境配置</li><ul><li>安装JDK，下载jdk-8u45-linux-x64.tar.gz，解压到/opt/jdk1.8.0_45</li></ul>           下载地址：<a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html" rel="nofollow" target="_blank">http://www.oracle.com/technetwork/java/javase/downloads/index.html</a><ul><li>安装scala,下载scala-2.11.6.tgz，解压到/opt/scala-2.11.6</li></ul>           下地地址： <a href="http://www.scala-lang.org/" rel="nofollow" target="_blank">http://www.scala-lang.org/</a><ul><li>安装Spark，下载spark-1.3.1-bin-hadoop2.6.tgz，解压到/opt/spark-hadoop</li></ul>                     下载地址：<a href="http://spark.apache.org/downloads.html," target="_blank">http://spark.apache.org/downloads.html,</a>                      <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/111810_E2Hc_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/111810_E2Hc_2248958.png" ></a><ul><li> </li><ul> 配置环境变量,编辑/etc/profile,执行以下命令     </ul></ul></ul>python<a href="http://my.oschina.net/u/555627" target="_blank" target="_blank">@ubuntu</a> :~$  sudo gedit /etc/profile                      在文件最增加：          #Seeting JDK JDK环境变量            export JAVA_HOME=/opt/jdk1.8.0_45            export JRE_HOME=${JAVA_HOME}/jre            export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib            export PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin:$PATH                  #Seeting Scala Scala环境变量            export SCALA_HOME=/opt/scala-2.11.6            export PATH=${SCALA_HOME}/bin:$PATH         #setting Spark Spark环境变量            export SPARK_HOME=/opt/spark-hadoop/        #PythonPath 将Spark中的pySpark模块增加的Python环境中          export PYTHONPATH=/opt/spark-hadoop/python<ul> 重启电脑，使/etc/profile永久生效，临时生效，打开命令窗口，执行 source /etc/profile  在当前窗口生效<li>测试安装结果</li><ul><li>打开命令窗口，切换到Spark根目录</li></ul></ul>      <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112031_fmoq_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112031_fmoq_2248958.png" ></a><ul><ul><li>执行 ./bin/spark-shell,打开Scala到Spark的连接窗口   </li></ul></ul>    <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112111_Br65_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112111_Br65_2248958.png" ></a>    <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112146_947n_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112146_947n_2248958.png" ></a>    启动过程中无错误信息，出现scala>，启动成功<ul><ul><li>   执行./bin/pyspark ,打开Python到Spark的连接窗口</li></ul></ul>    <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112222_v8Vk_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112222_v8Vk_2248958.png" ></a>                 <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112247_fGAH_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112247_fGAH_2248958.png" ></a> 启动过程中无错误，在出现如上所示时，启动成功。 <ul><ul><li>   通过浏览器访问：出现如下页面</li></ul></ul>    <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112318_agqv_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112318_agqv_2248958.png" ></a>              测试SPark可用。<ul><li>Python安发Spark应用</li><ul><li>前面已设置PYTHONPATH，将pyspark加入到Python的搜寻路径中</li><li>打开Spark安装目录，将Python-》build文件夹下py4j,复至到Python目录下，如图：</li></ul></ul>        <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112435_Nqqt_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112435_Nqqt_2248958.png" ></a><ul><ul><li>打开命令行窗口，输入python,Python版本为2.7.6，如图所示，注意Spark不支持Python3</li></ul></ul>        <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112515_nfpg_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112515_nfpg_2248958.png" ></a> <ul><ul><li>输入import pyspark，如下图所示，证明开发前工作完成</li></ul></ul>        <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112548_iXjk_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112548_iXjk_2248958.png" ></a><ul><ul><li>  使用Pycharm新建项项目，使用红框中的代码测试：</li></ul></ul>    <a class="example-image-link" href="http://static.oschina.net/uploads/space/2015/0521/112637_dgG3_2248958.png" data-lightbox="example-set"><img class="example-image" src="http://static.oschina.net/uploads/space/2015/0521/112637_dgG3_2248958.png" ></a></img>

spark

著作权归作者所有

如果觉得我的文章对您有用，请点赞。您的支持将鼓励我继续创作！

添加新评论0 条评论

Ctrl+Enter 发表

匿名评论

Ubuntu下Spark开发环境搭建

添加新评论0 条评论

作者其他文章

相关问题

相关资料