The Data Automation Imperative for Developers

By Fred Lherault, CTO EMEA and Emerging Markets at Pure Storage

Whether it’s an issue related to infrastructure, cost, security, accessibility, compatibility or scale, the manual management of data storage has always been a major challenge. Developers have struggled with this challenge for a long time.

By 2024, the annual amount of created, captured, copied, and consumed data is forecasted to cross 149 zettabytes. For enterprises — and particularly for developers, the explosion of unstructured data presents more than just a housekeeping problem.

The challenge of data growth

As organisations generate and acquire more data, these challenges are going to become more acute. Unstructured data — where most of the data growth is forecasted — typically requires file systems, directories or buckets to hold data. In the past, the systems that store this data have typically had limits in terms of scale and required a lot of feed and care which has been done by humans. This is not feasible anymore and modern data systems need to be able to scale transparently and non-disruptively and operate without human intervention.

Also, disk-based storage has traditionally been the default for the vast majority of data, but HDDs are already showing their limitations due to their excessive data center footprint, energy consumption, and overall costs required for unstructured data workloads. Modern all-Flash unstructured data storage platforms are able to fix all these challenges in a cost-effective manner. When combined with infrastructure and application automation, they enable organisations to benefit from the “Cloud operating model” anywhere.

The promise of storage automation for developers

Developers and application teams don’t need to know, or care, about the physical infrastructure behind their applications. They care about SLAs and having the proper resources to meet their performance, scale and data protection requirements. They need infrastructure that scales linearly and resources that are delivered instantly, with a few lines of code, an API call or a handful of clicks in a portal. They want the self-service cloud experience anywhere; and who can blame them?

Flexibility, agility, and speed to access are just as important to modern organisations as performance and reliability. Automation is crucial to addressing the volume and complexity of tedious storage management functions. As a first step though, businesses need to manage infrastructure through code instead of via manual processes.

Infrastructure as Code allows dev and ops teams to automatically manage, monitor, and provision resources without manually configuring hardware and operating systems. By splitting hardware and software infrastructure into modular components, developers can easily combine and automate them. Most importantly, they don’t need to worry about the configuration “drifting” over time through human errors. Modern Infrastructure as Code solutions ensure that the actual setup matches the requirements expressed through policies and classes of service. For developers, this means consistent results and fewer errors, manual deployments, and inconsistencies.

With automated Infrastructure as Code, developers can abstract the tedious management tasks in a multitude of ways, from handling initial workload placement with AI-driven recommendations to automatically adjusting and rebalancing the fleet when new resources come online. Users can self-service from a catalog of policy-based offerings while the infrastructure scales and adjusts itself to keep up with business demands.

This flexibility is key as organisations expand to hybrid-cloud models and adopt container orchestration tools like Kubernetes. To remain competitive, enterprises must support the new data services their customers demand. Developers want to consume Platform-as-a-Service and avoid cloud vendor lock-in, but often lack the operational expertise to manage multiple different services consistently. Modern Database-as-a-Service solutions enable consumption of different data services in any cloud through a single API call even if they don’t know how to deploy a given database, message queue or analytics solution.

Infrastructure as Code — the path to automation

To meet the needs of enterprise users and their customers, it’s imperative that data platforms moves towards “as Code” and automation. Infrastructure as code does come with challenges, which include merging new frameworks with existing technology and the complexity of migrations, but they are outweighed by major benefits such as the improved collaboration between IT teams, ease of automation and speed of delivery that it facilitates. Only by embracing Infrastructure as Code and automation can developers innovate at the pace needed by today’s organisations.