Source: Komprise
Intelligent data management is changing the way IT organisations view their data, data retention, ownership costs and positioning data for value and migrations. Data migrations have been a regular ongoing exercise as data growth requires larger systems and hardware continually ages out. But, they no longer need to be a complex burden.
Instead, a data migration is an opportunity to adopt modern architectures with systematic, continuous data movement that breaks the cycle of storage-focused hardware refreshes. Customers like Pfizer are saving 75% on storage by using Komprise to understand their unstructured data and then optimise data placement over cloud and on-premises resources.
In this blog, we will discuss best practices for smart data migrations and explain how Komprise enables easy, more intelligent data management, breaking the cycle of legacy storage hardware migrations.
When it comes to a Smart Data Migration, the Komprise mantra of “Know First, Move Smart, and Take Control” is being implemented by real-world customers with measurable benefits.
Know First (Gather Intel and Plan)
A successful data migration should be 90% preparation and 10% perspiration. Proper planning, understanding data’s value, growth, activity and stakeholder concerns are key to an easy, successful, smart data migration.
Smart Data Migration Tip: Communication with users, data owners and applications teams is vital to a successful migration. The migration goal and processes must be understood by all stakeholders. This leads to better decisions, faster migrations and fewer surprises. Professional services teams from leading enterprise storage companies use Komprise metrics to spot shares with rapid growth, identify issues and plan and execute migrations.
The journey from legacy storage management to intelligent data management begins with understanding your data and then working with data owners and stakeholders to align goals.
Komprise provides the intelligence on your unstructured data to make informed plans for data management and the movement of data. Simply point Komprise at your NFS, SMB and S3 sources and within 15 minutes, Komprise displays valuable analytics on all the data including how much is hot data, how much is cold data, how data is growing, and comparative data ownership costs.
Armed with this knowledge, storage teams can work with data owners to choose the optimal storage for their data. In this example, our legacy storage array hosts multiple shares and we can see a large percentage of the data is cold. This rarely-accessed data is a perfect candidate for cloud or on-premises object storage.
Smart Data Migration Tip: Using Komprise Intelligent Data Management and Transparent Move Technology™ ahead of a migration reduces data by 50 to 80 percent by tiering cold data first to cheaper, object storage. This significantly reduces the data movement and cutover times for a more efficient migration.
By coordinating with data owners, you can move data as entire shares, specific directories, or based on parameters like file type. The ability to segment data enables precise choices and reduces the size of migration tasks. This is an opportunity to move the right data to the optimal storage with colder data stored for durability and lower cost while hot data can be moved to flash for high performance.
Move Smart (and Quickly)
Komprise offers several ways to migrate faster and smarter.
Komprise Scale-out Performance
There are several components that will affect the speed at which data can move:
- Source storage performance
- Destination storage performance
- Network bandwidth between source and destination
- The number of data movers
- Numbers of files, average file sizes and quantity of directories
In large scale data migrations, the ability to parallelise the data across many nodes enables better utilisation of network bandwidth and the capabilities of the source and destination storage.
The Komprise Elastic Grid is designed to dynamically scale Observers and Windows Proxy Systems to optimise performance.
Optimised File Transfers
Standard file system clients were made to service end user and application requests—not perform massive data movements. Komprise optimises file transfers in many ways. First, by using analytics, Komprise can handle large and small files differently, thereby minimising network transfers over WAN. This reduces protocol chatter and leads to greater efficiency for large migrations.
Komprise Architecture
The Komprise Director is an intuitive user interface and management console:
- The Director gives a standard, comprehensive view of data across multiple systems and vendors.
- These analytics provide accessed last, file types, file sizes, who owns the data, and more.
- This visibility lets customers reposition data to where it can create value or reduce ownership costs.
- The Director is also a central migration management console where hundreds of migrations can be coordinated and administered.
The Komprise data movers are known as Observers, for NFS, and Windows Proxies, for SMB data. These simple virtual appliances form an elastic scale-out grid where you can add more nodes to increase throughput performance. The Komprise Grid is easy to scale up for a large initial data migration task and then later scale down to support ongoing data management or smaller migrations.
Move with Confidence
After identifying systems, reviewing data, consulting stakeholders, and building out the Komprise Grid, the next step is to confirm the plan and begin moving data.
Smart Data Migration Tip: Every file data migration plan should have these components:
- A tested cutover-plan. A clear set of steps must be in place to direct users and applications during the migration cut-over and seamlessly direct them to the new location.
- A verified fail-back plan. Equally as important as the cutover plan is knowing how to safely back off an active migration or cutover should an unexpected event occur.
With a well-formed and confirmed data migration plan in place it’s time to start the migration. The Komprise Director guides administrators through migration setup, prompting for the source and destination with options to maintain access time and full permissions.
Smart Data Migration Steps
- Once the migration tasks have been created, the migration runs continuous and automatic iterations that copy changes from the live source to the destination. This process is non-disruptive to users and applications accessing and updating data on the source.
- With each iteration, Komprise copies data from source to destination, including any new or updated files, while validating the integrity of the data by checking file checksums on both source and destination. The migration includes permissions and ACLs (access control list) to ensure identical security on the destination and to maintain access time for each file.
- Komprise logs any files that can’t be copied due to permission, file locking, or other issues. While some issues such as file locking will typically be resolved in later iterations, other issues such as permissions will require administrator intervention. Komprise maintains an advanced audit log to identify and help in resolving issues.
- After multiple iterations, Komprise identifies an average change or churn rate. This change rate gives an estimate of the amount of data that will need to be copied in the final migration iteration, and helps you plan the cutover.
- To prepare for the final iteration, the source must be placed in a read-only mode to prevent any file modification. This is a critical component of the migration plan where communication with the data owners is vital.
- During the final iteration, Komprise validates that all data on destination is identical to the source. Once the final iteration is complete the storage administration team will mark the migration complete, ending the active migration.
- The final step of the migration is to direct all users to the new storage and allow the appropriate permissions for production use.
Take Control with Continual Data Optimisation
The smart data migration began with an analysis of the data, identified the optimal resources, and then created a migration task to copy the data. Now we move from data migration to Intelligent Data Management. Komprise data management and data mobility policies continually analyze and transparently move data to the right place. What began as a migration task has evolved to continual optimization.
This is where infrastructure teams maintain their collaboration with the data owners to drive better business outcomes. Enterprise customers use Komprise Deep Analytics to create a Global File Index and engage with their lines of business. Data owners can create queries to identify data sets so the storage team can take actions like positioning data for more performance and tiering cold data to save costs without affecting end users.
Source: Komprise
Image source: Shutterstock (2127736745)
