(Created page with "== MapReduce MRUnit == MR Unit is a library for unit-testing Hadoop MapReduce jobs * https://mrunit.apache.org/ == Example == === Maven Dependency === For Hadoop2: <d...")
(No difference)

Latest revision as of 19:42, 20 November 2015

MapReduce MRUnit

MR Unit is a library for unit-testing Hadoop MapReduce jobs


Maven Dependency

For Hadoop2:


Use classifier hadoop1 for Hadoop1

This version works well with Hadoop 2.6.0-cdh5.4.7

Mapper Test


  • You wrap your mapper in MapDriver.newMapDriver
  • Feed some input data
  • Provide expected output
  • Run
mapper = MapDriver.newMapDriver(new YourMapper());

// input
mapper.addInput(key1, value1);
mapper.addInput(key2, value2);
mapper.addInput(key3, value3);

// expected output
mapper.addOutput(outKey1, outValue1);
mapper.addOutput(outKey2, outValue2);
mapper.addOutput(outKey3, outValue3);


Reducer Test

Same as for Mapper:

  • Wrap your reducer in ReduceDriver.newReduceDriver
  • Provide input and expected output
  • Run
reducer = ReduceDriver.newReduceDriver(new YourReducer());

reducer.addInput(key1, value1);
reducer.addInput(key2, value2);

reducer.addOutput(outKey1, outValue1);
reducer.addOutput(outKey2, outValue2);



Changing some configuration parameters is easy

  • use driver.getConfiguration()


reducer.getConfiguration().setInt("app.name.param1", 4);

Distributed Cache

Faking a file from "Distribute Cache":

URL resource = this.getClass().getResource("test-resource-file.txt");
Path link = Paths.get("symlink");
Path target = Paths.get(resource.toURI());
if (link.toFile().exists()) {

Files.createSymbolicLink(link, target);

try {
    // test goes here
} finally {