Categories

Jan 11 in Big Data | Hadoop

Q:

How will you monitor memory used in a Hadoop cluster?

1 Answer

Jan 11

In Hadoop, TaskTracker is the one that uses high memory to perform a task. We can configure the TastTracker to monitor memory usage of the tasks it creates. It can monitor the memory usage to find the badly behaving tasks, so that these tasks do not bring the machine down with excess memory consumption.

In memory monitoring we can also limit

 

the maximum memory used by a tasks. We can even limit the memory usage per node. So that all the tasks executing together on a node do not consume more memory than a limit.

Some of the parameters for setting memory monitoring in Hadoop are as follows:

mapred.cluster.map.memory.mb, mapred.cluster.reduce.memory.mb: This is the size of virtual memory of a single map/reduce slot in a cluster of Map-Reduce framework. mapred.job.map.memory.mb, mapred.job.reduce.memory.mb: This is the default limit of memory

 

set on each map/reduce task in Hadoop. mapred.cluster.max.map.memory.m mapred.cluster.max.reduce.memory This is the maximum limit of memory set on each map/reduce task in Hadoop.

Click here to read more about Loan/Mortgage
Click here to read more about Insurance

Related questions

Madanswer
Feb 23 in Big Data | Hadoop
Sep 7, 2019 in Big Data | Hadoop
Sep 7, 2019 in Big Data | Hadoop
...