0 votes
in Big Data | Hadoop by
How to compress mapper output in Hadoop?

1 Answer

0 votes
by
Mapper task processes each input record (from RecordReader) and generates a key-value pair and this key-value pairs generated by mapper is completely different from the input pair. The output of Mapper is also Known as intermediate output is written to the local disk.

To compress mapper output we should set conf.set(“mapreduce.map.output.compress”, true)

Apart from setting this property to enable compression for mapper output, we also need to consider some other factors like, which codec to use and what should be the compression type.

Following are the properties for configuring the same:-

mapred.map.output.compression.codec

mapred.output.compression.type

Out of these two factors, the choice of right codec is of more importance. As each codec has some pros and cons, you need to figure out, which suits your requirement. Generally, you would want faster read/write, a good compression factor and CPU friendly decompression (we have less number of reducers). Considering these factors, snappy codec feels like the best fit, as it has the faster read/write and a compression factor of 3.

Related questions

0 votes
asked Mar 27, 2020 in Big Data | Hadoop by AdilsonLima
0 votes
asked Feb 23, 2020 in Big Data | Hadoop by rahuljain1
...