Overseeing storage technologies has consistently posed a significant challenge for IT teams. Given that data is generated and confined within an organisation’s premises and frequently stored on various appliances, occasionally sourced from different providers, the traditional role of the data storage manager has leaned towards hardware proficiency. The primary emphasis lay on guaranteeing uninterrupted operation, overseeing installations, handling upgrades, procurement and seamless integrations.
According to Gartner’s projections, by 2026, major corporations will witness a threefold increase in their capacity for unstructured data. This expansion will encompass their on-premise setups, edge computing systems and public cloud infrastructures, a significant surge from the levels recorded in 2023.
At present, the prevailing paradigms for handling IT workloads centre around hybrid cloud architectures and edge computing. This shift has led to a dispersal of data storage, moving it beyond the confines of traditional corporate data centres.
The advent of Shadow IT, spurred by the proliferation of cloud solutions, has added a layer of complexity. Questions now arise regarding the extent of the organisation’s data ownership and its physical location.
Lastly, IT leaders are recognising the critical need to curate specific datasets from the expanse of their hybrid, multi-cloud storage, amounting to petabytes. These refined datasets are pivotal in supplying the burgeoning array of new AI tools with the requisite fuel for operation.
Instead of just overseeing storage, leaders in IT infrastructure are now tasked with the dual responsibility of managing data and providing essential data services. These services encompass safeguarding data, ensuring compliance, archiving, overseeing data lifecycles and expenses, and even overseeing data removal when it’s no longer required. Being accountable for data services also involves ensuring convenient access to the right data for end users and tools like cloud-based AI, all while keeping cost-efficiency in focus.
Defining Data Services in Data Management and Storage
The term “data services” encompasses a wide array of functions typically administered by enterprise IT. These include data processing, integration, security, reduction, protection, storage and the handling of unstructured data.
In the context of data storage and unstructured data management, data services encompass the full spectrum of data management, from its creation to its eventual retirement. This extends beyond primary storage and involves activities like analysing and reporting on storage growth and associated costs, including departmental allocations, monitoring data usage, facilitating self-service file search and tagging. Additionally, it encompasses data mobility functions such as migration, tiering, replication, and deletion. This modern approach necessitates the capability to comprehend data usage and administer data autonomously from the storage infrastructure.
A data services approach offers the following advantages:
- Comprehensive visibility and detailed search capabilities across various storage platforms and cloud environments.
- Analytical insights into data types and usage for making precise storage choices.
- Automation of policy-driven actions derived from this analysis.
- Mitigation of security and compliance vulnerabilities.
- Maximised utilisation of data, particularly in cloud environments.
- Empowerment of users with self-service access, catering to departmental and research requirements for data storage, administration, and AI workflows.
- Enhanced adaptability to embrace new storage, backup, and disaster recovery technologies, as data management remains vendor-agnostic.
Guidelines for Data Service Implementation
While there isn’t a definitive approach to adopting data services, the crux lies in analysis. Commencing with the utilisation of data analytics and management, comprehending data utilisation, growth and associated expenses across storage and cloud environments forms a solid foundation. Additionally, it’s imperative to empower data teams to categorise and locate data based on shared access permissions. These labels subsequently equip central IT with the ability to enact automated protocols, such as purging project files older than three years.
For IT infrastructure teams, a fresh set of metrics serve as a compass. While traditional metrics focused on parameters like latency, IOPS, network throughput, annual uptime and downtime, and backup windows, the new metrics shed light on the data itself.
These metrics encompass, but aren’t confined to:
- Principal data custodians
- Percentage of ‘inactive’ files untouched for a year or more
- Prevalent file sizes and formats
- Departmental storage expenses
- Vendor-specific storage costs per terabyte
- Reduced backup proportion
- Data expansion rate
Vigilance towards data anomalies is now paramount. Unusual activities, such as mass transfers or deletions of files, may signal a security breach. As organisations increasingly invest in AI, tools and procedures to track and document data inputs and outputs from these technologies become essential in mitigating privacy, security, legal, and IP risks.
Above all, experts in data management and storage infrastructure must pivot their mindsets and methods. Shifting from managing storage technologies to comprehending and overseeing data for diverse objectives is crucial. An infrastructure for data management and storage that champions adaptability and nimbleness in sync with organisational data requisites will expedite this transition, yielding superior outcomes for all stakeholders.
With a wealth of experience in networking and storage, IDS recognises that intelligent data management systems are the linchpin of any successful project. Whether organisations acknowledge it or not, unstructured data harbours untapped potential for pivotal organisational choices. Given that unstructured data is set to double every two years, Komprise steps in to help you manage this exponential growth while curbing expenses. If your business is considering a Komprise solution, reach out to our team at IDS today.
