A framework (abstraction) for processing and generating large datasets on a Distributed System

divide and conquer + Functional Programming + Pipelining



  • Input: kv pair
  • Output (emit): intermediate kv pair


After all maps have finished

  • Input: grouped intermediate kv pair (need to first collect and group before starting)
  • Output (emit): desired result


  • (In the old time) The bottle neck is in network, since GFS is involved, we can find who hold the input file then have the worker working on it.
  • Data streaming
  • Use Hashtable to group intermedia file
  • Load balance: Many more tasks than workers


  • MapReduce hides many details:
    • sending app code to servers
    • tracking which tasks are done
    • moving data from Maps to Reduces
    • balancing load over servers
    • recovering from failures