Jan 11, 2020 in Big Data | Hadoop
Q:

How will you monitor memory used in a Hadoop cluster?

1 Answer

0 votes
Jan 11, 2020

In Hadoop, TaskTracker is the one that uses high memory to perform a task. We can configure the TastTracker to monitor memory usage of the tasks it creates. It can monitor the memory usage to find the badly behaving tasks, so that these tasks do not bring the machine down with excess memory consumption.

In memory monitoring we can also limit

 

the maximum memory used by a tasks. We can even limit the memory usage per node. So that all the tasks executing together on a node do not consume more memory than a limit.

Some of the parameters for setting memory monitoring in Hadoop are as follows:

mapred.cluster.map.memory.mb, mapred.cluster.reduce.memory.mb: This is the size of virtual memory of a single map/reduce slot in a cluster of Map-Reduce framework. mapred.job.map.memory.mb, mapred.job.reduce.memory.mb: This is the default limit of memory

 

set on each map/reduce task in Hadoop. mapred.cluster.max.map.memory.m mapred.cluster.max.reduce.memory This is the maximum limit of memory set on each map/reduce task in Hadoop.

Related questions

+1 vote
Feb 23, 2020 in Big Data | Hadoop
0 votes
Jan 8, 2020 in Big Data | Hadoop
...