Amazon FSx for OpenZFS

Amazon FSx for OpenZFS, ‘a fully managed shared storage built on the popular OpenZFS file system’, was launched in AWS Sydney Region (ap-southeast-2) in April 2022.

Powered by Graviton processors (AWS designed processors that as per AWS delivering a major leap in performance and capabilities and providing the best price performance), Amazon FSx for OpenZFS enables a range of new capabilities, such as higher performance, lower latency, broad access by Linux, Windows and MacOS operating systems, all at a lower price point against other comparable shared storage services offered by AWS.

OpenZFS file system has its root from the ZFS file system, which was developed by Sun Microsystems over twenty years ago, in 2001, for the proprietary Solaris operating system. ZFS was designed with considerations for large, powerful servers at the time, which use-case focus was leaning more towards performance, functionality and reliability, rather than economical costing.

A few years later in 2005, Sun Microsystems released OpenSolaris under CDDL (Common Development and Distribution License), making the Solaris source code public. Logically, that included ZFS source code. Since then, ports of ZFS to other platforms and operating systems have been gathering momentum, enabled by developers over the world who are attracted by the features and functionalities that ZFS file system could provide. OpenZFS was born. (For those who had interest to know, Sun Microsystems was acquired by Oracle in 2010 which discontinued OpenSolaris, making Solaris close source again. To ZFS enthusiasts, it felt almost as if OpenSolaris escaped out for a few short years to give birth to OpenZFS.)

Some features that OpenZFS storage may provide include:

  • Protection against data corruption. Integrity checking for both data and metadata
  • Continuous integrity verification and automatic “self-healing” repair
  • Data redundancy with mirroring
  • Support for high storage capacities — up to 256 trillion yobibytes (2^128 bytes)
  • Storage space saving with transparent compression using LZ4, GZIP or ZSTD
  • Hardware-accelerated native encryption
  • Efficient storage with snapshots and copy-on-write clones
  • Efficient local or remote replication — send only changed blocks with ZFS send and receive

Amazon FSx for OpenZFS is AWS’ implementation of a fully managed OpenZFS file system based storage service. It provides the familiar features, performance, and capabilities of OpenZFS file systems with the agility, scalability and simplicity benefits of cloud native services.

Amazon FSx for OpenZFS provides access to shared file storage over the latest versions of the NFS protocol (v3, v4, v4.1, v4.2).

High-speed, low-latency and outstanding IOPS performance are signature characteristics of Amazon FSx for OpenZFS. The table below highlighted its Throughput and IOPS capabilities:


Comparison with Amazon FSx for Windows File Server

AWS has been offering an FSx service for years – the Amazon FSx for Windows File Server, which enables a shared storage service in AWS cloud for Windows based applications. (Despite the name and the fact that there are many benefits for Windows based applications to use FSx for shared storage, Amazon FSx for Windows File Server can also be accessed by Linux and MacOS operating systems.)

Built on Windows Server, full support for the SMB protocol, Windows NTFS and DFS, and Active Directory (AD) integration are some of the key characteristics of Amazon FSx for Windows File Server.

Did I just mention AD? Here comes probably the foremost difference between the two AWS FSx services. Amazon FSx for Windows File Server enjoys AD based identity authentication and AD based access control which can go down to the level of individual files and folders.

Amazon FSx for OpenZFS, on the other hand, does not rely on, and as such does not have AD integration. Logically, there is no AD based access control to files and folders in FSx for OpenZFS. It uses AWS IAM for OpenZFS API call authentication with identities. It supports POSIX based individual file and folder access permissions.

We can see that whether the workload utilises AD based file / folder access and permission will have a big say on which FSx service to use.

As to the performance, Amazon FSx for OpenZFS indeed offers higher throughput, higher IOPS, lower latency, and lower pricing. The advantage becomes greater and greater should the demand on both technical performance and price performance gets higher and higher – the more IOPS and throughput capacities the workload demands, the better that FSx for OpenZFS would perform comparing to FSx for Windows File Server.

There is another ‘however’ – Amazon FSx for OpenZFS is currently offered in single AZ (Availability Zone) model only. Though it does have built-in high availability within the AZ, the availability SLA is the single-AZ’s 99.5%. On the other hand, Amazon FSx for Windows File Server offers both single-AZ and multi-AZ models – providing the option to spend more for the multi-AZ option to get that 99.9% SLA.

The table below provides a summary on the comparison between the Amazon FSx for OpenZFS service and the Amazon FSx for Windows File Server service.


                                                                                            *Table contents were abstracted from AWS documentation.

Like anything in modern clouds, services and technologies continue to evolve. Amazon FSx for OpenZFS offers cutting edge performance benefits in cloud native shared storage that surely will interest many customers and use cases. There are also unique features that Amazon FSx for Windows File Server service, and indeed other flavours of cloud storage services, provide. Their best use cases depend on the workload characteristics. Altogether, cloud native storage services offer comprehensive applications and future proof service assurance.

                                                                                                                                            -- Simon Wang

Comments

Post a Comment

Popular posts from this blog

AWS Storage Gateway File Gateway with S3 and FSx For Lustre with S3

Image
Cloud data storage is widely used not just by cloud native compute environments but also on-premises computer systems and applications. It has been so as there are tremendous benefits that cloud data storage brings, most of which are difficult or impossible for non-cloud data solutions. These include but not limited to: • Virtually unlimited capacity • The ease of provisioning • Being on-demand • Data durability (through the distribution and replication powered by the cloud) • Cost efficiency • Reduced or eliminated maintenance overhead AWS Storage Gateway is a common storage service for on-premises and AWS cloud hybrid environments – whether it is being in the transition phase of a migration from on-premises to the cloud or that both on-premises and cloud environments will continue to work together and coexist for the times to come. AWS Storage Gateway provides local caching for on-premises application’s latency-improved access to the cloud data storage.   Various protocol
Read more

Fairness Evaluation and Model Explainability In AI

Image
Artificial Intelligence (AI) relies on machine learning and its modelling to provide outcomes. Imagine a credit card company receives hundreds of thousands of applications each year and would like to use AI to do the first round of application filtering. If it is not developed properly, the AI decisions will be skewed with unfairness – such as rejecting a lot more applications from a certain age group, from a certain gender, or from a certain employment history pattern – UNFAIRLY. Please note unfairness is the key issue here – in other words, AI and machine learning inference that is not reflecting the due course in actual situations. This is typically called bias in AI and machine learning. Biases can come from various stages of a machine learning life cycle: • Biases may exist in the pre-training data -  e.g., the dataset to be used in machine learning training has biases in it • Biases may be introduced by the machine learning exercise  Below is a machine learning lifecycle char
Read more

AWS and Generative AI

Image
 Generative Artificial Intelligence, or Generative AI, GenAI, is one of the hottest topics in the recent months. You would ‘have to be living under a rock to not notice this massive topic’.  ‘Generative AI is a branch of AI that focuses on creating new data. It is a subset of machine learning. The goal of generative AI is to create new data that is similar to the data that was used to train the model.’ – texts generated by an AI tool.  Recently I experimented a couple of random text-to-image GenAI tools to observe how generative AI can create contents and how the data generated by different GenAI tools trained by different models is different. My experiment was straightforward: text-to-image GenAI tools ask you to provide some texts as a prompt, and then use this input to generate an image. So that was what I did. The prompt I supplied was: ‘female student graduation photo in front of a white building with flowers in hand in Paris’. Both GenAI tools I tested could output an image withi
Read more