Installing Apache Hadoop 3.4.2 on Linux 2026 (Single Node Cluster) Complete Step-by-Step Guide

S1. Update the System Before installing Hadoop, update your system packages. sudo apt...

Dev.to · Jul 29, 2026

Installing Apache Hadoop 3.4.2 on Linux 2026 (Single Node Cluster) Complete Step-by-Step Guide

S1. Update the System

Before installing Hadoop, update your system packages.

sudo apt update
sudo apt upgrade -y

Enter fullscreen mode Exit fullscreen mode

S2. Install and Configure Java 21

Apache Hadoop is written in Java and requires a compatible Java Runtime Environment (JRE) and Java Development Kit (JDK).

This guide uses OpenJDK 21 (LTS).

Java Compatibility

Installed Java Version	Action
Java 21	Continue with the installation
Java 17	May work with some Hadoop versions, but this guide uses Java 21
Java 22–25	Install Java 21 and switch to it
Java Not Installed	Install OpenJDK 21

S2.1 Check if Java is Installed

java -version
javac -version

Enter fullscreen mode Exit fullscreen mode

If Java is NOT Installed

Install OpenJDK 21.

sudo apt update
sudo apt install openjdk-21-jdk -y

Enter fullscreen mode Exit fullscreen mode

Verify the installation.

java -version
javac -version

Enter fullscreen mode Exit fullscreen mode

If Java is Already Installed

Check the installed version.

java -version
javac -version

Enter fullscreen mode Exit fullscreen mode

Compare the output with the compatibility table above.

If Java 21 is already active, continue to the next step.

Otherwise, switch to Java 21.

S2.2 List All Installed Java Versions

update-java-alternatives -l

Enter fullscreen mode Exit fullscreen mode

Example:

java-1.17.0-openjdk-amd64
java-1.21.0-openjdk-amd64

Enter fullscreen mode Exit fullscreen mode

S2.3 Switch to Java 21

sudo update-alternatives --config java
sudo update-alternatives --config javac

Enter fullscreen mode Exit fullscreen mode

Select the option corresponding to Java 21.

S2.4 Verify the Active Java Version

java -version
javac -version

Enter fullscreen mode Exit fullscreen mode

Both commands should display Java 21.

S2.5 Find the Correct JAVA_HOME

Different Linux distributions may install Java in different directories.

Instead of copying someone else's path, determine it automatically.

readlink -f $(which java)

Enter fullscreen mode Exit fullscreen mode

Example:

/usr/lib/jvm/java-21-openjdk-amd64/bin/java

Enter fullscreen mode Exit fullscreen mode

Therefore,

JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64

Enter fullscreen mode Exit fullscreen mode

Note: Never blindly copy the JAVA_HOME path from a tutorial. Always determine it using the command above.

S3. Download Apache Hadoop 3.4.2

Move to the directory where you want to download Hadoop.

cd ~/Downloads

Enter fullscreen mode Exit fullscreen mode

Download Hadoop from the official Apache Archive.

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.4.2/hadoop-3.4.2.tar.gz

Enter fullscreen mode Exit fullscreen mode

Verify the download.

ls -lh hadoop-3.4.2.tar.gz

Enter fullscreen mode Exit fullscreen mode

Expected output:

-rw-r--r-- 1 user user 734M hadoop-3.4.2.tar.gz

Enter fullscreen mode Exit fullscreen mode

S4. Extract and Install Hadoop

After downloading the Hadoop archive, the next step is to extract it and install Hadoop on your system.

In this guide, Hadoop will be installed under /opt/hadoop, which is the standard location for optional third-party software on Linux.

S4.1 Extract the Archive

Move to the directory where the archive was downloaded.

cd ~/Downloads

Enter fullscreen mode Exit fullscreen mode

Extract the archive.

tar -xvzf hadoop-3.4.2.tar.gz

Enter fullscreen mode Exit fullscreen mode

This will create a directory named:

hadoop-3.4.2

Enter fullscreen mode Exit fullscreen mode

Verify the extraction.

ls

Enter fullscreen mode Exit fullscreen mode

Expected output:

hadoop-3.4.2
hadoop-3.4.2.tar.gz

Enter fullscreen mode Exit fullscreen mode

S4.2 Move Hadoop to /opt

Move the extracted directory to /opt.

sudo mv hadoop-3.4.2 /opt/hadoop

Enter fullscreen mode Exit fullscreen mode

S4.3 Verify the Installation Directory

Check that Hadoop has been moved successfully.

ls /opt

Enter fullscreen mode Exit fullscreen mode

Expected output:

hadoop

Enter fullscreen mode Exit fullscreen mode

View the contents of the Hadoop directory.

ls /opt/hadoop

Enter fullscreen mode Exit fullscreen mode

Expected output:

bin
etc
include
lib
libexec
licenses
sbin
share
NOTICE.txt
README.txt
...

Enter fullscreen mode Exit fullscreen mode

S4.4 Change Ownership

By default, the Hadoop directory is owned by the root user because it was moved using sudo.

Change the ownership so your user can modify the configuration files without requiring root privileges every time.

sudo chown -R $USER:$USER /opt/hadoop

Enter fullscreen mode Exit fullscreen mode

Verify the ownership.

ls -ld /opt/hadoop

Enter fullscreen mode Exit fullscreen mode

Example output:

drwxr-xr-x <username> <username> ...

Enter fullscreen mode Exit fullscreen mode

S4.5 Verify the Hadoop Version

Run the Hadoop executable directly.

/opt/hadoop/bin/hadoop version

Enter fullscreen mode Exit fullscreen mode

Expected output:

Hadoop 3.4.2
Source code repository ...
Compiled by ...

Enter fullscreen mode Exit fullscreen mode

At this point, Hadoop has been successfully installed on the system. However, it cannot yet be executed from any directory because its binaries have not been added to the system's PATH.

Note: We are intentionally using the full path (/opt/hadoop/bin/hadoop) for now. In the next section, we will configure environment variables so that the hadoop, hdfs, and yarn commands can be run from anywhere in the terminal.

S5. Configure Environment Variables

After installing Hadoop, we need to configure environment variables. These variables tell the operating system where Hadoop and Java are installed, allowing Hadoop commands to be executed from any directory.

The main environment variables are:

Variable	Purpose
`JAVA_HOME`	Points to the Java installation directory
`HADOOP_HOME`	Points to the Hadoop installation directory
`HADOOP_CONF_DIR`	Specifies the Hadoop configuration directory
`PATH`	Allows Hadoop commands to be executed from anywhere

S5.1 Check Your Default Shell

Before editing your shell configuration, determine which shell you are using.

echo $SHELL

Enter fullscreen mode Exit fullscreen mode

Possible outputs:

/bin/bash

Enter fullscreen mode Exit fullscreen mode

/bin/zsh

Enter fullscreen mode Exit fullscreen mode

S5.2 Open the Shell Configuration File

If you are using Bash

nano ~/.bashrc

Enter fullscreen mode Exit fullscreen mode

If you are using Zsh

nano ~/.zshrc

Enter fullscreen mode Exit fullscreen mode

S5.3 Add the Hadoop Environment Variables

Append the following lines to the end of the file.

# Java
export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))

# Hadoop
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

# Hadoop Components
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME

# Native Libraries
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

# PATH
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Enter fullscreen mode Exit fullscreen mode

Why use `$(dirname $(dirname $(readlink -f $(which java))))`?

Instead of hardcoding the Java installation path, this command automatically determines the correct JAVA_HOME directory. This makes the guide more portable across different Linux distributions and Java installation methods.

S5.4 Save the File

For Nano:

Press Ctrl + O
Press Enter
Press Ctrl + X

S5.5 Reload the Configuration

Bash

source ~/.bashrc

Enter fullscreen mode Exit fullscreen mode

Zsh

source ~/.zshrc

Enter fullscreen mode Exit fullscreen mode

S5.6 Verify the Environment Variables

Check that Hadoop is configured correctly.

echo $JAVA_HOME

Enter fullscreen mode Exit fullscreen mode

Example:

/usr/lib/jvm/java-21-openjdk-amd64

Enter fullscreen mode Exit fullscreen mode

Check the Hadoop installation directory.

echo $HADOOP_HOME

Enter fullscreen mode Exit fullscreen mode

Expected output:

/opt/hadoop

Enter fullscreen mode Exit fullscreen mode

Check the configuration directory.

echo $HADOOP_CONF_DIR

Enter fullscreen mode Exit fullscreen mode

Expected output:

/opt/hadoop/etc/hadoop

Enter fullscreen mode Exit fullscreen mode

S5.7 Verify Hadoop Commands

You should now be able to run Hadoop commands from any directory.

Check the Hadoop version.

hadoop version

Enter fullscreen mode Exit fullscreen mode

Check the HDFS version.

hdfs version

Enter fullscreen mode Exit fullscreen mode

Check the YARN version.

yarn version

Enter fullscreen mode Exit fullscreen mode

Each command should report Hadoop 3.4.2.

Note: If any of these commands return command not found, ensure that the shell configuration file has been reloaded using the source command or open a new terminal session.

S6. Configure Passwordless SSH

Apache Hadoop uses SSH to start and stop its daemons. Even in a single-node cluster, passwordless SSH is required for Hadoop scripts such as start-dfs.sh and start-yarn.sh.

S6.1 Check if OpenSSH is Installed

Run the following command:

ssh -V

Enter fullscreen mode Exit fullscreen mode

If OpenSSH is installed, you will see output similar to:

OpenSSH_10.x

Enter fullscreen mode Exit fullscreen mode

If the command is not found, install the OpenSSH server.

sudo apt update
sudo apt install openssh-server -y

Enter fullscreen mode Exit fullscreen mode

S6.2 Start and Enable the SSH Service

Start the SSH service.

sudo systemctl start ssh

Enter fullscreen mode Exit fullscreen mode

Enable it to start automatically on boot.

sudo systemctl enable ssh

Enter fullscreen mode Exit fullscreen mode

Check the service status.

sudo systemctl status ssh

Enter fullscreen mode Exit fullscreen mode

The status should show active (running).

Exit the status screen by pressing Q.

S6.3 Generate an SSH Key Pair

If you do not already have an SSH key, generate one.

ssh-keygen -t rsa -b 4096

Enter fullscreen mode Exit fullscreen mode

When prompted for the file location, simply press Enter to accept the default location.

When prompted for a passphrase, press Enter twice to leave it empty.

Example:

Enter file in which to save the key:
/home/<username>/.ssh/id_rsa

Enter fullscreen mode Exit fullscreen mode

S6.4 Enable Passwordless SSH

Append the public key to the authorized keys file.

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Enter fullscreen mode Exit fullscreen mode

Set the correct permissions.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Enter fullscreen mode Exit fullscreen mode

S6.5 Test the SSH Connection

Run:

ssh localhost

Enter fullscreen mode Exit fullscreen mode

The first time you connect, you may see:

Are you sure you want to continue connecting (yes/no)?

Enter fullscreen mode Exit fullscreen mode

Type:

yes

Enter fullscreen mode Exit fullscreen mode

If everything is configured correctly, you will be logged into your own machine without entering a password.

Exit the SSH session.

exit

Enter fullscreen mode Exit fullscreen mode

Why is Passwordless SSH Required?

Hadoop's management scripts use SSH to launch and stop services such as the NameNode, DataNode, ResourceManager, and NodeManager. Without passwordless SSH, these scripts will repeatedly prompt for a password, making automated startup impossible.

S7. Create Hadoop Data Directories

Hadoop requires a few directories to store its metadata and temporary files. These directories are referenced by the Hadoop configuration files.

Create the required directories by running the following commands:

mkdir -p ~/hadoopdata/namenode
mkdir -p ~/hadoopdata/datanode
mkdir -p ~/hadooptmp

Enter fullscreen mode Exit fullscreen mode

You can verify that the directories were created successfully using:

tree -L 2 ~

Enter fullscreen mode Exit fullscreen mode

If the tree command is not installed, install it using:

sudo apt install tree -y

Enter fullscreen mode Exit fullscreen mode

Expected output:

/home/<username>
├── hadoopdata
│   ├── datanode
│   └── namenode
└── hadooptmp

Enter fullscreen mode Exit fullscreen mode

Note: The directory names must match the paths specified in the Hadoop configuration files (core-site.xml and hdfs-site.xml). If you changed those paths, create the directories accordingly.

S8. Install the Hadoop Configuration Files

Instead of manually editing multiple Hadoop configuration files, use the installation script provided in the GitHub repository. The script downloads and installs all the required configuration files automatically.

Download the installation script.

wget https://raw.githubusercontent.com/itsadityapidurkar/hadoop-installation-guide/main/linux/install-configs.sh

Enter fullscreen mode Exit fullscreen mode

Make the script executable.

chmod +x install-configs.sh

Enter fullscreen mode Exit fullscreen mode

Run the script.

./install-configs.sh

Enter fullscreen mode Exit fullscreen mode

The script downloads and installs the following configuration files:

core-site.xml
hdfs-site.xml
mapred-site.xml
yarn-site.xml
workers

These files are automatically copied to:

/opt/hadoop/etc/hadoop

Enter fullscreen mode Exit fullscreen mode

If the script completes successfully, you should see:

========================================
 Apache Hadoop Configuration Installer
========================================

[+] Downloading configuration files...
[✓] Download complete.

[+] Installing configuration files...

[✓] Configuration files installed successfully!

Installed to:
/opt/hadoop/etc/hadoop

Enter fullscreen mode Exit fullscreen mode

Note: The provided configuration files are intended for a Single Node Hadoop 3.4.2 installation. If you modified the directory paths while following this guide, update the configuration files accordingly before installing them.

S9. Configure `hadoop-env.sh`

The hadoop-env.sh file is used to configure environment variables required by Hadoop. The only change required for this guide is to set the JAVA_HOME variable.

Open the file using a text editor.

nano /opt/hadoop/etc/hadoop/hadoop-env.sh

Enter fullscreen mode Exit fullscreen mode

Search for the line containing:

export JAVA_HOME=

Enter fullscreen mode Exit fullscreen mode

# export JAVA_HOME=

Enter fullscreen mode Exit fullscreen mode

Replace it with:

export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))

Enter fullscreen mode Exit fullscreen mode

Save the file and exit.

For Nano:

Press Ctrl + O
Press Enter to save the file.
Press Ctrl + X to exit.

You can verify that the JAVA_HOME path is detected correctly by running:

echo $(dirname $(dirname $(readlink -f $(which java))))

Enter fullscreen mode Exit fullscreen mode

Example output:

/usr/lib/jvm/java-21-openjdk-amd64

Enter fullscreen mode Exit fullscreen mode

Note: This command automatically detects the Java installation directory, making the configuration portable across different Linux distributions and Java installation paths.

S10. Format the NameNode

Before starting Hadoop for the first time, the NameNode must be formatted. This initializes the Hadoop Distributed File System (HDFS) and creates the required metadata.

Run the following command:

hdfs namenode -format

Enter fullscreen mode Exit fullscreen mode

During the formatting process, you may be prompted with the following message if the NameNode directory already contains data:

Re-format filesystem in Storage Directory root=... ? (Y or N)

Enter fullscreen mode Exit fullscreen mode

Type:

Enter fullscreen mode Exit fullscreen mode

and press Enter.

If the formatting is successful, the last few lines of the output should look similar to:

...
INFO common.Storage: Storage directory ... has been successfully formatted.
INFO namenode.FSImageFormatProtobuf: Saving image file ...
INFO namenode.NameNode: SHUTDOWN_MSG:

Enter fullscreen mode Exit fullscreen mode

Important: Formatting the NameNode permanently removes all existing HDFS metadata and data stored in the configured NameNode and DataNode directories. This command should only be executed during the initial setup or when you intentionally want to reset the Hadoop cluster. Do not run this command every time you start Hadoop.

S11. Start HDFS

Now that the NameNode has been formatted, you can start the Hadoop Distributed File System (HDFS).

Run the following command:

start-dfs.sh

Enter fullscreen mode Exit fullscreen mode

If HDFS starts successfully, you should see output similar to:

Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [<hostname>]

Enter fullscreen mode Exit fullscreen mode

To verify that all HDFS daemons are running, execute:

jps

Enter fullscreen mode Exit fullscreen mode

Expected output:

NameNode
DataNode
SecondaryNameNode
Jps

Enter fullscreen mode Exit fullscreen mode

Note: The order of the processes may vary depending on your system.

You can also verify that the NameNode is running by opening the following URL in your web browser:

http://localhost:9870

Enter fullscreen mode Exit fullscreen mode

If HDFS has started successfully, the NameNode web interface will be displayed, showing information about the cluster, storage usage, live DataNodes, and the HDFS filesystem.

S12. Start YARN

After starting HDFS, the next step is to start YARN (Yet Another Resource Negotiator), which is responsible for resource management and job scheduling in the Hadoop cluster.

Run the following command:

start-yarn.sh

Enter fullscreen mode Exit fullscreen mode

If YARN starts successfully, you should see output similar to:

Starting resourcemanager
Starting nodemanagers

Enter fullscreen mode Exit fullscreen mode

To verify that the YARN daemons are running, execute:

jps

Enter fullscreen mode Exit fullscreen mode

Expected output:

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager
Jps

Enter fullscreen mode Exit fullscreen mode

Note: The order of the processes may vary depending on your system.

You can also verify that YARN is running by opening the following URL in your web browser:

http://localhost:8088

Enter fullscreen mode Exit fullscreen mode

If YARN has started successfully, the ResourceManager web interface will open, displaying information about the cluster, available resources, running applications, and node status.

At this point, your Hadoop Single Node Cluster is up and running.

S13. Verify the Hadoop Installation

After starting both HDFS and YARN, verify that all Hadoop services are running correctly.

Check the running Hadoop daemons.

jps

Enter fullscreen mode Exit fullscreen mode

Expected output:

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager
Jps

Enter fullscreen mode Exit fullscreen mode

View the HDFS cluster report.

hdfs dfsadmin -report

Enter fullscreen mode Exit fullscreen mode

If the cluster is running correctly, the report will display information about the configured capacity, remaining storage, live DataNodes, and other cluster details.

Next, open the NameNode Web UI in your browser.

http://localhost:9870

Enter fullscreen mode Exit fullscreen mode

The NameNode dashboard should display information such as:

Cluster Summary
Live DataNodes
Storage Information
HDFS Overview

Then open the ResourceManager Web UI.

http://localhost:8088

Enter fullscreen mode Exit fullscreen mode

The ResourceManager dashboard should display:

Cluster Metrics
Node Status
Running Applications
Available Resources

If all the above commands and web interfaces are accessible, your Hadoop Single Node Cluster has been installed and configured successfully.

Hadoop Web UI Port Quick-Reference

Service	Web UI URL	Default Port	Description
HDFS NameNode	`http://localhost:9870`	9870	File system status, capacity, browser directories
YARN ResourceManager	`http://localhost:8088`	8088	Job executions, node resource allocation
HDFS DataNode	`http://localhost:9864`	9864	Individual DataNode status & blocks

S14. Shut Down the Hadoop Cluster

To stop the Hadoop daemons and release system resources safely, stop the YARN resource manager and HDFS services in order.

Run the following commands:

stop-yarn.sh
stop-dfs.sh

Enter fullscreen mode Exit fullscreen mode

To verify that all Hadoop services have been stopped successfully, run:

jps

Enter fullscreen mode Exit fullscreen mode

Expected output:

Jps

Enter fullscreen mode Exit fullscreen mode

Only the Jps process itself should be running, confirming that HDFS and YARN have successfully shut down.

Read the original source Next news

Installing Apache Hadoop 3.4.2 on Linux 2026 (Single Node Cluster) Complete Step-by-Step Guide

S1. Update the System

S2. Install and Configure Java 21

Java Compatibility

S2.1 Check if Java is Installed

If Java is NOT Installed

If Java is Already Installed

S2.2 List All Installed Java Versions

S2.3 Switch to Java 21

S2.4 Verify the Active Java Version

S2.5 Find the Correct JAVA_HOME

S3. Download Apache Hadoop 3.4.2

S4. Extract and Install Hadoop

S4.1 Extract the Archive

S4.2 Move Hadoop to /opt

S4.3 Verify the Installation Directory

S4.4 Change Ownership

S4.5 Verify the Hadoop Version

S5. Configure Environment Variables

S5.1 Check Your Default Shell

S5.2 Open the Shell Configuration File

If you are using Bash

If you are using Zsh

S5.3 Add the Hadoop Environment Variables

Why use $(dirname $(dirname $(readlink -f $(which java))))?

S5.4 Save the File

S5.5 Reload the Configuration

Bash

Zsh

S5.6 Verify the Environment Variables

S5.7 Verify Hadoop Commands

S6. Configure Passwordless SSH

S6.1 Check if OpenSSH is Installed

S6.2 Start and Enable the SSH Service

S6.3 Generate an SSH Key Pair

S6.4 Enable Passwordless SSH

S6.5 Test the SSH Connection

Why is Passwordless SSH Required?

S7. Create Hadoop Data Directories

S8. Install the Hadoop Configuration Files

S9. Configure hadoop-env.sh

S10. Format the NameNode

S11. Start HDFS

S12. Start YARN

S13. Verify the Hadoop Installation

Hadoop Web UI Port Quick-Reference

S14. Shut Down the Hadoop Cluster

0 Comments

Why use `$(dirname $(dirname $(readlink -f $(which java))))`?

S9. Configure `hadoop-env.sh`