Blog

ZFS – the ultimate storage platform

We used to talk in kilobytes in the age of narrow band, we used to have 9.6 Kbps modems to connect to the internet, and so was the speed for our fax machines, we later migrated to Megabytes, with the complete storage size of our personal computers / desktops, a little later we started talking of a fast ethernet speed of 100 Mbps, upgraded of 700 Mbps with e-CAT-5 and just a few years back it became de-facto Gigabit networks. but that was cVolume of datacommunication and this is storage.

The storage requirement is growing with leaps and bounds, in today’s tech savvy world. in before internet era, everything was printed and people needed physical space for storing the files, the concept changed with the advent of internet. Internet created a cyberworld, and data became digitised. data moved from print files to soft files, transfer of data became easier in a digital format. and then came social media, & smart phones, that digitized our lives. Now everything around us is data.

you take picture from your phone – it’s data, you record video – it’s data, you record voice – that’s data too and then what you watch on YouTube, check on WhatApp, send SMS, chat, Facebook, TED, Vimeo, TV Serials, even this blog that you are reading  – is all data.

With such huge amount of data, we need huge storage too, and with huge storage, come huge and complex methodology for management and making the right data available at the right time. the old system is obsolete now. This created a need for a new file system designed to handle such huge data volumes spread over multiple media storage in billions and billions of files.

Let us understand the data terminology before we proceed further in understanding the new file system.

kB to Yb

This was foreseen by Sun Microsystems in year 2005 itself and they developed ZFS (Zettabyte File System)over their earlier system called NFS (Network File System). Although when developed, it was proprietary for Sun Microsystem, but now it is Open Sourced, with OpenSolaris Forum. Eventually Sun was bought over by Oracle, who distanced themselves from OpenSolaris Forum. The OpenSolaris Community / forum along with the breakaway employees of Sun, after the Sun’s merger with Oracle, formed illomos.org . illomos is the OS that includes ZFS. Various people use illomos as the platform to build their own solution

ZFS differentiator philosophy

ZFS is a file system combined with a volume management system, designed for high storage capacities. The features include protection against data corruption, efficient data compression, snapshots and copy-on-write clones, continuous integrity checking and automatic repair.

ZFS has three major design goals:Zettabytes

  • Data Search & Management: In order to manage and search data on Zettabyte size of storage, we need to have a reference mechanism for marking where, what is stored. In computing, a cache is the mechanism (hardware or software) that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation, or the duplicate of data stored elsewhere. Therefore, multiple caching mechanisms provide increased performance.
    • Adaptive Replacement Cache (ARC) is an advanced memory-based read cache system provided in ZFS.
    • A second level of disk-based read cache can be added with L2ARC (Level 2, ARC), and
    • Yet another disk-based synchronous write cache is made available in ZFS called ZIL (ZFS Intent Log).
  • Virtualization & Orchestration: Physical storage devices are added to a pool, and storage space is allocated from that shared pool. Space is available to all file systems, and can be increased by adding new storage devices to the pool.
  • Data Correction & Replication: All data includes a checksumof the data. When data is written, the checksum is calculated and written along with it. When that data is later read back, the checksum is calculated again. If the checksums do not match, a data error has been detected. ZFS will attempt to automatically correct errors when data redundancy is available.

ZFS is significantly different from any previous file system because it is more than just a file system. Combining the traditionally separate roles of volume manager and file system provides ZFS with unique advantages. The file system is now aware of the underlying structure of the disks. Traditional file systems could only be created on a single disk at a time. If there were two disks then two separate file systems would have to be created.

In a traditional hardware RAID configuration, this problem was avoided by presenting the operating system with a single logical disk made up of the space provided by a number of physical disks, on top of which the operating system placed a file system. ZFS’s combination of the volume manager and the file system solves this and allows the creation of many file systems all sharing a pool of available storage.

Comments (1)

  1. Asadur Rahman / Reply August 28, 2016 at 3:42 am

    nice

Leave a Comment