Big Data/Google File System
Sources[edit | edit source]
- IBM General Parallel FS
- Soft Updates: A Solution to the Metadata Update Problem in File Systems
- QFS = faster version of GFS
Problem[edit | edit source]
Need Storage Server that supports
- many 100TB of data
- distributed along many 1000 servers of cheap hardware
- serving many 100 of clients.
- Build and store Google search index
- Store and analyze Log Files of Production servers and Google Analytics
- YouTube (!)
- Research computations
Many of those tasks can be based on map reduce.
Existing Solutions[edit | edit source]
RAID RAID Redundant array of inexpensive/independent disks offers block level access to multiple hard drives attached to a single machine.
SAN Storage Area Network Dedicated High Speed Network that connect storage nodes over ATA/SCSI. Typically over FiberChannel or Ethernet.
Distributed FS Distributed FS
API Levels[edit | edit source]
Block Level API for communication via ATA or SCSI between the disc controller and the CPU offers access to the individual blocks of the hard drive. There is no file system abstraction given. Block devices have to be formatted before use. API Description?
The File System API offered by the OS / device driver.
File Level API offered by protocols like FTP/SMB or NFS.
Further reading: Techrepublic.com
Specifications[edit | edit source]
Run on cluster of commodity servers. Serves clients in the same network.
Offer simplified File Level API to clients
- Open / Close file specified by path
- Read file at arbitrary locations (fast)
- Write file at arbitrary locations (may be slow)
- Append to file (fast)
- Scalability: Automatic adjust when new nodes join the cluster
- Fault Tolerance: Deal with hardware failures
- Concurrency: Allow multiple clients to access the same files at the same time.
- Locality: Perform computations locally if possible
- Concurrent appends should be fast
- High throughput (at cost of high latency)
Baseline Ideas[edit | edit source]
- Mount all devices at a single machine via e.g. NFS. Expose to clients via FTP interface.
- Big RAID at one machine
- Storage Area Network