| brew install snappy |
| CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ant compile-native |
| chmod 644 build/native/Mac_OS_X-x86_64-64/lib/* |
| ls -l /usr/local/lib/libsnappy.dylib |
| ls -l build/native/Mac_OS_X-x86_64-64/lib/libhadoop.dylib |
I am using Hadoop in Mac OS X. I think I installed Hadoop properly. When I tried to run an example WordCount problem, I got this error: INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable. Setting up Hadoop 2.6 on Mac OS X Yosemite. After comparing different guides on the internet, I ended up my own version base on the Hadoop official guide with manual download. If you prefer Homebrew, this one would be your best choice. Actually there is no difference in the configuration of these two methods except the file directories.
Hadoop Native Library Mac Os X 10
搜索后发现已经有人给出了Mac OSX 系统下编译生成本地库的patch，下面详细介绍在Mac OSX 平台下编译Hadoop本地库的方法。 一、环境说明： Hadoop 2.2.0; Mac OS X 10.9.1. 另外，在 Mac OS X 10.7 有个 Bug，见 Hadoop logs errors upon startup on OS X 10.7，报错信息为 『Unabel to load realm info from SCDynamicStore』，不知是否修复。解决该问题需要设置 Hadoop 环境变量中的 HADOOPOPTS ：. Oct 04, 2018 How to Install Hadoop on Mac. Now let’s move further to the procedure of installation of Hadoop on Mac OS X. Installing Hadoop on Mac is not as simple as typing a single line command in Terminal to perform an action. It requires a mix of knowledge, concentration and patience. However, you don’t need to worry about not knowing everything. This patch makes the autoconf script work on Mac OS X. LZO needs to be installed (including the optional shared libraries) for the compile to succeed. You'll want to regenerate the configure script using autoconf after applying this patch.
Jan 03, 2017 Install Spark on Mac (PySpark). While the native hadoop library is supported on Linux type platforms only (The library does not to work with Cygwin or the Mac OS X.
| commit a5681a94b83e7314a46863f811226e4e51967595 |
| Author: David Phillips <david@acz.org> |
| Date: Wed Oct 10 18:34:24 2012 -0700 |
| Compile native libraries on Mac OS X |
| diff --git a/src/native/Makefile.am b/src/native/Makefile.am |
| index c70de7d..6268877 100644 |
| --- a/src/native/Makefile.am |
| +++ b/src/native/Makefile.am |
| @@ -33,9 +33,7 @@ export PLATFORM = $(shell echo $$OS_NAME | tr [A-Z] [a-z]) |
| AM_CPPFLAGS = @JNI_CPPFLAGS@ -I$(HADOOP_NATIVE_SRCDIR)/src |
| -Isrc/org/apache/hadoop/io/compress/zlib |
| - -Isrc/org/apache/hadoop/io/compress/snappy |
| - -Isrc/org/apache/hadoop/io/nativeio |
| - -Isrc/org/apache/hadoop/security |
| + -Isrc/org/apache/hadoop/io/compress/snappy |
| AM_LDFLAGS = @JNI_LDFLAGS@ -m$(JVM_DATA_MODEL) |
| AM_CFLAGS = -g -Wall -fPIC -O2 -m$(JVM_DATA_MODEL) |
| @@ -43,13 +41,7 @@ lib_LTLIBRARIES = libhadoop.la |
| libhadoop_la_SOURCES = src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c |
| src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c |
| src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c |
| - src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c |
| - src/org/apache/hadoop/security/getGroup.c |
| - src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c |
| - src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c |
| - src/org/apache/hadoop/io/nativeio/file_descriptor.c |
| - src/org/apache/hadoop/io/nativeio/errno_enum.c |
| - src/org/apache/hadoop/io/nativeio/NativeIO.c |
| + src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c |
| libhadoop_la_LDFLAGS = -version-info 1:0:0 $(AM_LDFLAGS) |
| libhadoop_la_LIBADD = -ldl -ljvm |
| diff --git a/src/native/acinclude.m4 b/src/native/acinclude.m4 |
| index 266ac87..48481b3 100644 |
| --- a/src/native/acinclude.m4 |
| +++ b/src/native/acinclude.m4 |
| @@ -13,8 +13,10 @@ AC_CACHE_CHECK([Checking for the 'actual' dynamic-library for '-l$1'], ac_cv_lib |
| ac_cv_libname_$1='`objdump -p conftest | grep NEEDED | grep $1 | sed 's/W*NEEDEDW*(.*)W*$/'1'/'`' |
| elif test ! -z '`which ldd | grep -v 'no ldd'`'; then |
| ac_cv_libname_$1='`ldd conftest | grep $1 | sed 's/^[[[^A-Za-z0-9]]]*([[[A-Za-z0-9.]]]*)[[[^A-Za-z0-9]]]*=>.*$/'1'/'`' |
| + elif test ! -z '`which otool | grep -v 'no otool'`'; then |
| + ac_cv_libname_$1='`otool -L conftest | grep $1 | sed -e 's/^[ ]*//' -e 's/ .*//' -e 's/.*/(.*)$/1/'`'; |
| else |
| - AC_MSG_ERROR(Can't find either 'objdump' or 'ldd' to compute the dynamic library for '-l$1') |
| + AC_MSG_ERROR(Can't find either 'objdump', 'ldd' or 'otool' to compute the dynamic library for '-l$1') |
| fi |
| else |
| ac_cv_libname_$1=libnotfound.so |
| diff --git a/src/native/configure.ac b/src/native/configure.ac |
| index 71ff66d..87b34a1 100644 |
| --- a/src/native/configure.ac |
| +++ b/src/native/configure.ac |
| @@ -56,7 +56,7 @@ JNI_LDFLAGS=' |
| if test $JAVA_HOME != ' |
| then |
| JNI_LDFLAGS='-L$JAVA_HOME/jre/lib/$OS_ARCH/server' |
| - JVMSOPATH=`find $JAVA_HOME/jre/ -name libjvm.so | head -n 1` |
| + JVMSOPATH=`find $JAVA_HOME/jre/ -name 'libjvm.*' | head -n 1` |
| JNI_LDFLAGS='$JNI_LDFLAGS -L`dirname $JVMSOPATH`' |
| fi |
| ldflags_bak=$LDFLAGS |