Map-Reduce Ready Distributed File System
First Claim
1. A map-reduce compatible shuffle function, comprising:
- a distributed file system; and
a map-reduce system, wherein each map function writes to the distributed file system and each reduce function reads input from the distributed file system.
7 Assignments
0 Petitions
Accused Products
Abstract
A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Key-value stores relate keys to data for such purposes as directories, container location maps, and offset maps in compressed files.
165 Citations
24 Claims
-
1. A map-reduce compatible shuffle function, comprising:
-
a distributed file system; and a map-reduce system, wherein each map function writes to the distributed file system and each reduce function reads input from the distributed file system. - View Dependent Claims (2, 3, 4)
-
-
5. A map-reduce compatible distributed file system, comprising:
-
a container location database (CLDB) configured to maintain information about where each of a plurality of containers is located; a plurality of cluster nodes, each cluster node containing one or more storage pools, each storage pool containing zero or more containers; and a plurality of inodes for structuring data within said containers. - View Dependent Claims (6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)
-
Specification