hadoop2.2.0 在 windows 上安装 分类： hapoop2013-12-30 20:28 2480 人阅读 评论(5) 收藏 目录(?)[+] Good news for Hadoop developers who want to use Microsoft Windows OS for their development activities. Finally Apache Hadoop 2.2.0 release officially supports for running Hadoop on Microsoft Windows as well. But the bin distribution of Apache Hadoop 2.2.0 release does not contain some windows native components (likewinutils.exe, hadoop.dll etc). As a result, if we try to run Hadoop in windows, we'll encounter ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path In this article, I'll describe how to build bin native distribution from source codes, install, configure and run Hadoop in Windows Platform. Tools and Technologies used in this article : 1. 2. 3. 4. 5. 6. 7. Apache Hadoop 2.2.0 Source codes Windows 7 OS Microsoft Windows SDK v7.1 Maven 3.1.1 Protocol Buffers 2.5.0 Cygwin JDK 1.6 Build Hadoop bin distribution for Windows 1. 2. 3. 4. 5. Download and install Microsoft Windows SDK v7.1 . Download and install Unix command-line tool Cygwin . Download and install Maven 3.1.1 . Download Protocol Buffers 2.5.0 and extract to a folder (say c:\protobuf ). Add Environment Variables JAVA_HOME , M2_HOME and Platform if not added already.
Add Environment Variables: Note : Variable name Platform is case sensitive. And value will be either x64 orWin32 for building on a 64-bit or 32-bit system. Edit Path Variable to add bin directory of Cygwin (say C:\cygwin64\bin ), bindirectory of Maven (say C:\maven\bin ) and installation path of Protocol Buffers(say c:\protobuf ). Edit Path Variable:
6. Download hadoop-2.2.0-src.tar.gz and extract to a folder having short path (sayc:\hdfs ) to avoid runtime problem due to maximum path length limitation in Windows. Select Start --> All Programs --> Microsoft Windows SDK v7.1 and open Windows SDK 7.1 Command Prompt . Change directory to Hadoop source code folder (c:\hdfs ). Execute mvn package with options -Pdist,native-win -DskipTests -Dtarto create Windows binary tar distribution. Windows SDK 7.1 Command Prompt Setting SDK environment relative to C:\Program Files\Microso ft SDKs\Windows\v7.1\. Targeting Windows 7 x64 Debug 7.
C:\Program Files\Microsoft SDKs\Windows\v7.1>cd c:\hdfs C:\hdfs>mvn package -Pdist,native-win -DskipTests -Dtar [INFO] Scanning for projects... [INFO] ----------------------------------------------------------------------[INFO] Reactor Build Order: [INFO] [INFO] Apache Hadoop Main [INFO] Apache Hadoop Project POM [INFO] Apache Hadoop Annotations [INFO] Apache Hadoop Assemblies [INFO] Apache Hadoop Project Dist POM [INFO] Apache Hadoop Maven Plugins [INFO] Apache Hadoop Auth [INFO] Apache Hadoop Auth Examples [INFO] Apache Hadoop Common [INFO] Apache Hadoop NFS [INFO] Apache Hadoop Common Project [INFO] Apache Hadoop HDFS [INFO] Apache Hadoop HttpFS
[INFO] Apache Hadoop HDFS BookKeeper Journal [INFO] Apache Hadoop HDFS-NFS [INFO] Apache Hadoop HDFS Project [INFO] hadoop-yarn [INFO] hadoop-yarn-api [INFO] hadoop-yarn-common [INFO] hadoop-yarn-server [INFO] hadoop-yarn-server-common [INFO] hadoop-yarn-server-nodemanager [INFO] hadoop-yarn-server-web-proxy [INFO] hadoop-yarn-server-resourcemanager [INFO] hadoop-yarn-server-tests [INFO] hadoop-yarn-client [INFO] hadoop-yarn-applications [INFO] hadoop-yarn-applications-distributedshell [INFO] hadoop-mapreduce-client [INFO] hadoop-mapreduce-client-core [INFO] hadoop-yarn-applications-unmanaged-am-launcher [INFO] hadoop-yarn-site
[INFO] hadoop-yarn-project [INFO] hadoop-mapreduce-client-common [INFO] hadoop-mapreduce-client-shuffle [INFO] hadoop-mapreduce-client-app [INFO] hadoop-mapreduce-client-hs [INFO] hadoop-mapreduce-client-jobclient [INFO] hadoop-mapreduce-client-hs-plugins [INFO] Apache Hadoop MapReduce Examples [INFO] hadoop-mapreduce [INFO] Apache Hadoop MapReduce Streaming [INFO] Apache Hadoop Distributed Copy [INFO] Apache Hadoop Archives [INFO] Apache Hadoop Rumen [INFO] Apache Hadoop Gridmix [INFO] Apache Hadoop Data Join [INFO] Apache Hadoop Extras [INFO] Apache Hadoop Pipes [INFO] Apache Hadoop Tools Dist [INFO] Apache Hadoop Tools