Category: HadoopEco

Add nodes to DFS/YARN cluster

Mingmin

In previous post I created a HDFS/YARN cluster with only 4 nodes. Now I’ve requested more resource, it’s time to extend the capacity.

Here I plan to add another 4 hosts to the cluster, both HDFS and YARN. Please node that if the host profile(mostly RAM) is not the same, you should change yarn.nodemanager.resource.memory-mb in etc/hadoop/yarn-site.xml properly.

1. environment setup

First of all, let’s setup the environment for each host.

a. enable passwordless access for new added hosts;

b. copy hadoop-2.7.4.tar.gz, jdk-7u71-linux-x64.tar.gz and jdk-8u144-linux-x64.tar.gz to all hosts, and unzip them;

tar -zxf hadoop-2.7.4.tar.gz ; 
tar -zxf jdk-7u71-linux-x64.tar.gz ; 
tar -zxf jdk-8u144-linux-x64.tar.gz;

c. create local directories

 mkdir -p /mnt/dfs/namenode /mnt/dfs/data /mnt/yarn/nm-local-dir /mnt/yarn/nm-log /mnt/dfs/journal;  
chown -R stack:stack /mnt/dfs/ /mnt/yarn

d. add profile /etc/profile.d/hadoop.sh

HADOOP_HOME=/home/stack/hadoop-2.7.4
export HADOOP_HOME
HADOOP_PREFIX=/home/stack/hadoop-2.7.4
export HADOOP_PREFIX
export HADOOP_CONF_DIR=/home/stack/hadoop-2.7.4/etc/hadoop
export YARN_CONF_DIR=/home/stack/hadoop-2.7.4/etc/hadoop

2. DFS/YARN configuration

Since all my hosts have the same size, I only need to add new host list to etc/hadoop/slaves and copy all configurations to new hosts.

3. enable DataNode/NodeManager services

$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

$HADOOP_HOME/sbin/yarn-daemon.sh --config $HADOOP_HOME/etc/hadoop start nodemanager

Now you should see the new nodes in both HDFS and YARN cluster.

setup a YARN cluster with HA

Mingmin1 Comment

Recently, I’m creating a YARN cluster for POC projects to run BeamSQL with FlinkRunner. Since it’s built on cloud VMs, HA is a critical feature to avoid single-point-failure, for both DFS NameNode and YARN ResourceManager.

The latest version that is supported in Flink is Hadoop® 2.7, so I decide to go with hadoop-2.7.4.

1. environment setup

1.1. host list

In the initial cluster, I have
* 2 hosts for DFS Namenode;
* 2 hosts for YARN ResourceManager;
* 4 hosts to act as both DFS DataNode and YARN NodeManager, –3 of them are used as JournalNode as well;
* 3 hosts to for a shared Zookeeper;

1.2. environment settings

a) create the same account and config passwordless access between each other.

It’s suggested to have different accounts, dfs for HDFS services and yarn for YARN services.

b) install JDK
Although the Hadoop wiki says that JDK7 is supported, but I see some error when starting the services which asks for JDK8. So I would install JDK8 here. –I also install JDK7 which is used in Flink.

2. Hadoop settings

2.1. /etc/profile.d/hadoop.sh

A system level profile /etc/profile.d/hadoop.sh is added to include environment parameters.

HADOOP_HOME=/home/stack/hadoop-2.7.4
export HADOOP_HOME
HADOOP_PREFIX=/home/stack/hadoop-2.7.4
export HADOOP_PREFIX
export HADOOP_CONF_DIR=/home/stack/hadoop-2.7.4/etc/hadoop
export YARN_CONF_DIR=/home/stack/hadoop-2.7.4/etc/hadoop

2.2. etc/hadoop/hadoop-env.sh

export JAVA_HOME=/home/stack/jdk1.8.0_144
export HADOOP_CONF_DIR=/home/stack/hadoop-2.7.4/etc/hadoop
export HADOOP_HEAPSIZE=2048

2.3. etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://dfs1</value>
    </property>
</configuration>

2.4. etc/hadoop/hdfs-site.xml

<configuration>
    <!-- HA -->
    <property>
      <name>dfs.nameservices</name>
      <value>dfs1</value>
    </property>
    <property>
      <name>dfs.ha.namenodes.dfs1</name>
      <value>nn1,nn2</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.dfs1.nn1</name>
      <value>dfs-nm.example.com:8020</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.dfs1.nn2</name>
      <value>dfs-nm-ha.example.com:8020</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.dfs1.nn1</name>
      <value>dfs-nm.example.com:50070</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.dfs1.nn2</name>
      <value>dfs-nm-ha.example.com:50070</value>
    </property>
    <property>
      <name>dfs.namenode.shared.edits.dir</name>
      <value>qjournal://nm1.example.com:8485;nm2.example.com:8485;nm3.example.com:8485/toradfs1</value>
    </property>
    <property>
      <name>dfs.client.failover.proxy.provider.toradfs1</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
      <name>dfs.ha.fencing.methods</name>
      <value>shell(/bin/true)</value>
    </property>
    <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/mnt/dfs/journal/node/local/data</value>
    </property>
    <property>
      <name>dfs.ha.automatic-failover.enabled</name>
      <value>true</value>
    </property>
    <property>
      <name>ha.zookeeper.quorum</name>
      <value>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</value>
    </property>

    <!-- namenode -->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/mnt/dfs/namenode</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.blocksize</name>
        <value>268435456</value>
    </property>
    <property>
        <name>dfs.namenode.handler.count</name>
        <value>100</value>
    </property>

    <!-- datanode -->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/mnt/dfs/data</value>
    </property>
</configuration>

2.5. etc/hadoop/yarn-site.xml

<configuration>
    <!-- ResourceManager -->
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>256</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>8192</value>
    </property>

    <!-- NodeManager -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>40960</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>2.1</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/mnt/yarn/nm-local-dir</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/mnt/yarn/nm-log</value>
    </property>
    <!-- health check -->
    <property>
        <name>yarn.nodemanager.health-checker.script.path</name>
        <value>/home/stack/hadoop-2.7.4/etc/hadoop/health_check.sh</value>
    </property>
    <property>
        <name>yarn.nodemanager.health-checker.interval-ms</name>
        <value>600000</value>
    </property>


    <!-- HistoryServer -->
    <!-- YARN HA -->
    <property>
      <name>yarn.resourcemanager.ha.enabled</name>
      <value>true</value>
    </property>
    <property>
      <name>yarn.resourcemanager.cluster-id</name>
      <value>yarn1</value>
    </property>
    <property>
      <name>yarn.resourcemanager.ha.rm-ids</name>
      <value>rm1,rm2</value>
    </property>
    <property>
      <name>yarn.resourcemanager.hostname.rm1</name>
      <value>yarn-rm1.example.com</value>
    </property>
    <property>
      <name>yarn.resourcemanager.hostname.rm2</name>
      <value>yarn-rm2.example.com</value>
    </property>
    <property>
      <name>yarn.resourcemanager.webapp.address.rm1</name>
      <value>yarn-rm1.example.com:8088</value>
    </property>
    <property>
      <name>yarn.resourcemanager.webapp.address.rm2</name>
      <value>yarn-rm2.example.com:8088</value>
    </property>
    <property>
      <name>yarn.resourcemanager.zk-address</name>
      <value>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</value>
    </property>
</configuration>

2.6. etc/hadoop/workers

nm1.example.com
nm2.example.com
nm3.example.com
nm4.example.com

3. initialize ans start DFS/YARN cluster

3.1. create local folders

a. create directory /mnt/dfs/namenode in the two NameNodes;
b. create directory /mnt/dfs/journal in the three JouunalNode;
c. create directory /mnt/dfs/data, /mnt/yarn/nm-local-dir and /mnt/yarn/nm-log in the 4 DataNode/NodeManager;

3.2. start DFS cluster

Start DFS is a bit tricky, there’re some back-and-forth steps.
a. start journalnode in all hosts

$HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode

b. format both 2 namenodes

$HADOOP_PREFIX/bin/hdfs namenode -format TORA_YARN1

c. format zkfc in one namenode host

$HADOOP_PREFIX/bin/hdfs zkfc -formatZK

d. stop journalnode in all hosts

$HADOOP_HOME/sbin/hadoop-daemon.sh stop journalnode

e. start the cluster

$HADOOP_PREFIX/sbin/start-dfs.sh
$HADOOP_PREFIX/bin/hdfs dfsadmin -report

3.3. start YARN cluster

a. start services in ResourceManager

$HADOOP_HOME/sbin/yarn-daemon.sh --config $HADOOP_HOME/etc/hadoop start resourcemanager

b. start services in NodeManager

$HADOOP_HOME/sbin/yarn-daemon.sh --config $HADOOP_HOME/etc/hadoop start nodemanager

3.4. check cluster status

You can now go to the web UI to verify both NameNode and ResourceManager are running in HA mode. For NameNode, (active) is shown in the active NameNode, and (standby) is shown in the standby NameNode. For ResourceManager, if you open the URL for standby node, it’s redirected to active node automately.

HDFS/YARN UI:
http://yarn-rm1.example.com:8088/cluster
http://dfs-nm.example.com:50070/dfshealth.html#tab-overview
http://dfs-nm-ha.example.com:50070/dfshealth.html#tab-overview

Refer links:
1. http://hadoop.apache.org/docs/r2.7.4/hadoop-project-dist/hadoop-common/ClusterSetup.html
2. http://hadoop.apache.org/docs/r2.7.4/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
3. http://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html