GettingStartedWithHadoop

Refer to GettingStartedWithHadoop – Hadoop Wiki.

Below, I’ll paste the settings in my test system. It’s a one node system running in the virtualbox. Since it’s a little tricky to sync the time between milti VM OS, I just install one for test.

Part I. System Configuration

1.  IP setting:

hadoop@hadoop:~$ less /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.56.101
netmask 255.255.255.0
network 192.168.56.0
broadcast 192.168.56.255
gateway 192.168.56.1
2. host/ip
hadoop@hadoop:~$ less /etc/hosts
127.0.0.1       localhost
127.0.1.1       hadoop.corp.ebay.com    hadoop
192.168.56.101  hadoop.corp.ebay.com    hadoop
3. profile setting
hadoop@hadoop:~$ less /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).
...

export JAVA_HOME=/home/hadoop/java
export JRE_HOME=/home/hadoop/java/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

export HADOOP_HOME=/home/hadoop/hadoop
export PATH=$HADOOP_HOME/bin:$PATH

export HBASE_HOME=/home/hadoop/hbase
export PATH=$HBASE_HOME/bin:$PATH

Part II. Hadoop Configuration
Changed Files under $HADOOP_HOME/conf
-rw-r--r-- 1 hadoop hadoop  261 2011-08-18 01:24 mapred-site.xml
-rw-r--r-- 1 hadoop hadoop  416 2011-08-18 01:21 core-site.xml
-rw-r--r-- 1 hadoop hadoop  265 2011-08-18 01:18 hdfs-site.xml
-rw-r--r-- 1 hadoop hadoop    7 2011-08-18 01:15 slaves
-rw-r--r-- 1 hadoop hadoop    7 2011-08-18 01:14 masters
-rw-r--r-- 1 hadoop hadoop 2.3K 2011-08-18 01:14 hadoop-env.sh
4. hadoop-env.sh
++
# The java implementation to use.  Required.
export JAVA_HOME=/home/hadoop/java
5. masters & slaves
hadoop@hadoop:~/hadoop/conf$ cat masters slaves
hadoop
hadoop
6. core-site.xml
hadoop@hadoop:~/hadoop/conf$ less core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://hadoop:9000</value>
</property>

</configuration>
7. hdfs-site.xml
hadoop@hadoop:~/hadoop/conf$ less hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property> <!-- This is for hbase -->
  <name>dfs.datanode.max.xcievers</name>
  <value>4086</value>
</property>

</configuration>
8. mapred-site.xml
hadoop@hadoop:~/hadoop/conf$ less mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
<name>mapred.job.tracker</name>
<value>hadoop:9001</value>
</property>

</configuration>