Quick trick update python to version 2.7 on Centos 6.5

yum install centos-release-SCL
yum install python27

Then if you want to use if in your shell you would run something like:

scl enable python27 bash

Disclaimer: this is a trick for using python 2.7 in a separate environment, without removing 2.6 because it would break Yum on this centos version.

Heads up: starting hadoop

if you see this when you starting or stoping hadoop:


[hadoop@dev ~]$ start-dfs.sh 
13/10/25 22:21:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now....... 
sed: -e expression #1, char 6: unknown option to `s' HotSpot(TM): ssh: 
Could not resolve hostname HotSpot(TM): Name or service not known 64-Bit: ssh: 
Could not resolve hostname 64-Bit: 
Name or service not known 

the workaround is to set the following environment variables: 
export HADOOP_HOME=/usr/local/hadoop 
export PATH=$HADOOP_HOME/bin:$PATH
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Cheat sheet for Hadoop , up, up !!!!

[root@hulk sbin]# hadoop version
Hadoop 2.4.1
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1604318
Compiled by jenkins on 2014-06-21T05:43Z
Compiled with protoc 2.5.0
From source with checksum bb7ac0a3c73dc131f4844b873c74b630
This command was run using /usr/local/hadoop-2.4.1/share/hadoop/common/hadoop-common-2.4.1.jar

Check hadoop is running
[root@hulk sbin]# hadoop dfsadmin -report
List the content of home directory
$ hdfs dfs -ls /user/claudio
Upload file from local file to HDFS
$ hdfs dfs -put songs.txt /user/claudio
Cat the content of the file from HDFS
$ hdfs dfs -cat /user/claudio/songs.txt
Change permissions to file
$ hdfs dfs -chmod 700 /user/claudio/songs.txt
Set the replication factor of the file to 4
$ hdfs dfs -setrep -w 4 /user/claudio/songs.txt
Check the file size
$ hdfs dfs -du -h /user/adam/songs.txt
Create subdirectory in your home directory
$ hdfs dfs -mkdir songs
Move files across
$ hdfs dfs -mv songs.txt songs/
Remove directory from HDFS
$ hdfs dfs -rm -r songs