Frequent readers of this blog know that I am obsessed with data protection in general and data restoration specifically.  Obviously these two elements are critical for today’s data-intensive businesses and there are a multitude of vendors providing solutions to address these challenges.  It can be difficult to assess the benefits of a given approach and the concepts of Recovery Time Objective(RTO) and Restore Point Objective(RPO) are useful metrics to consider when analyzing the benefits of different solutions.  In this blog entry, I will discuss these two measures and why they are relevant to your organization.

Recovery Time Objective

This is a critical metric for illustrating the risk of potential downtime.  SNIA defines the term as follows:

The maximum acceptable time period required to bring one or more applications and associated data back from an outage to a correct operational state

Thus RTO is a measure of how fast data can be recovered.  As you can imagine, there are a range of options for reducing RTO and in general, the shorter the RTO, the higher the cost.  Common solutions for RTO reduction include, disk-based backup, CDP technology and array-based snapshots.  Each of those approaches bring a benefits in RTO, but may add capital expense and operational complexity.

The other element to consider is that not all applications should be treated the same.  For example, RTO requirements on a mission-critical Oracle database are likely to be very different from those on filers holding personal data.  In the former case, a lengthy outage could dramatically impact the business while in the latter, it could cause annoyance with a minimal business impact.

When considering a solution you should look at its RTO metrics and analyze how they align with your business objectives.  Additionally, remember that RTO and product cost are often inversely related and so while a short RTO might be nice for all applications, the resulting costs may not meet your budgetary requirements.

Recovery Point Objective

RPO addresses the granularity of recovery and the frequency of backup.  SNIA defines it as follows:

The maximum acceptable time period prior to a failure or disaster during which changes to data may be lost as a consequence of recovery.

The question to ask yourself is, “how much data can I afford to lose if an outage were to occur.”  In the traditional nightly backup model, you are potentially at risk of losing up to 24 hours worth of data.  You could reduce the RPO by performing more frequent backups during the day.  Alternative approaches include CDP or snapshots.  However, these can add cost and complexity.

Like RTO, RPO requirements can vary by application.  Critical applications with large change rates may require shorter RPOs versus infrequently changing lower tier applications.  Meeting these divergent needs typically requires different approaches to creating data backups.  However, in general the more granular the RPO, the more expensive the solution.  Thus, RPO must be reviewed in the context of application criticality and budget.

Summary

RTO and RPO are critical metrics for today’s data protection environments, and choosing the appropriate approach requires a detailed understanding of the environment and business objectives.  It is often useful to combine solutions to achieve RTO and RPO requirements while managing total costs.  A classic example is utilizing frequently daily snapshots for short-term RPO and RTO while still relying on full backups nightly for longer term retention.  The full backups enable you to limit the number of snapshots retained which can reduce costs and environment complexity.  A SEPATON solution can address these challenges by improving RTO in the traditional backup process and potentially improving RPO by allowing for less impactful intraday backups.  It will provide these benefits at a cost that is much less than alternative approaches.

No related posts.

I started this blog over two years ago to focus on the criticality of data protection and specifically data recovery.  While technology continues to evolve, the importance of these two elements remains consistent.  Every company must have a recovery strategy to protect against data loss or corruption.  Some people may be inclined to de-emphasize backup and recovery based on the faulty assumption that today’s virtualized hardware and software is more reliable or flexible, but this is a mistake.  In the last month, we have seen two examples of why data recovery is critical, and both affected entities had large IT staffs and huge budgets.  Without an effective protection strategy, massive data loss would have been unavoidable in both cases.  The companies recovered the vast majority of their data but experienced an outage that was far longer and more damaging than either anticipated.

The first situation occurred at the Virginia Information Technology Agency (VITA), the organization that provides IT for the state of Virginia.  Jon Toigo has a detailed post on the event and the agency has a site dedicated to the outage.  In summary, on August 25, 2010 a critical disk array experienced a problem which was compounded by human error and resulted in total data loss. VITA was forced to initiate a complete recovery from physical tape, and a week later, the system was not fully online.  Eventually, VITA successfully recovered about 97% of their data and the remaining 3% may be gone forever.  The combination of data loss and an extended outage reflected poorly on VITA, their subcontractors and suppliers, and resulted in the Governor of Virginia calling for a third-party investigation.

The second situation impacted Chase Bank’s online banking systems and occurred on Monday, September 13.  During the 48 hour outage, Chase’s online banking systems were inaccessible resulting in extreme customer frustration.  The Register has an article on the subject. To summarize, the database software used for the online banking system experienced an error resulting in unrecoverable data corruption.  The same corruption was replicated to the hot backup making it unusable as well and so Chase was forced to recover from their offline media.  They restored the full backup from Saturday and rolled forward the transaction logs.  The result was a complete recovery albeit with longer then desired downtime.

Both of these examples serve as important reminders that data loss can occur at any time.  The sources of these outages can include hardware, software and human error and is often a combination of these.  When losses occur, the resumption of operations depends on the ability to rapidly recover data.  Both VITA and Chase suffered from slower then desired recovery times and extended outages.  Clearly, the longer the downtime the greater the business impact and in both cases, the entities have had to deal with strong backlash from their constituencies.

Extended outages like these must be avoided.  In order to assess restore risk, end users should closely analyze their recovery time objectives (RTO) and ensure that they choose technology that aligns with their requirements.  A high performance disk-based backup solution like a SEPATON S2100-ES2 can assist in this area by providing recovery speeds of up to 17.3 TB/hr and the reliability of a highly redundant architecture.  Users must also remember that recovery tests are critical and should perform complete restore tests periodically.  The goal is to ensure that RTO requirements are met and that the appropriate critical data is being protected.  The combination of these two strategies will not prevent outages, but ensure that a company is prepared when one occurs.

Related posts:

1. Lessons from the Sidekick debacle
2. Protecting personal data

My last blog post contained a poll asking visitors about their primary VMware backup methodology.  The survey listed the common approaches to protecting virtualized environments including traditional agent-based,  VCB/VADP, dedicated VMware backup application, snapshots and doing nothing.  The results suggest that that the agent-based approach is most commonly used.  I anticipate that end users will migrate to backup methodologies that support VMware’s VADP functionality, but believe that there will always be a subset of people who rely on the agent-based approach. When implementing the agent-based approach, you should consider the following:

Benefits of Agent-Based Backup Models:

Transparent to existing processes – The agent-based approach is exactly the same as the traditional physical server model and so most backup administrators are familiar with the process.  With this approach, each VM has a separate client backup license and transfers its to the backup server over the LAN.

Granular restores – Each VM is backed up at the file level.  Therefore, standard incremental and/or full backup policies apply and file-level recoveries are possible regardless of OS.

Application consistency – Backup vendors provide a variety of server agents that ensure that specific applications (like Exchange, SQL or Oracle) are quiesced prior to backup.  These same agents can be used inside the VMs to ensure that running applications are in a consistent state during backup.  A consistent backup ensures the fastest and most reliable recoveries.

Challenges of Agent-Based Backup Models:

System resources – The process of backing up a VM can be I/O and CPU intensive – a situation that is exacerbated by backing up multiple VMs on the same host simultaneously.  The agent-based approach creates the challenge of managing the backup process to minimize simultaneous VM backups.  The proliferation of VMs can make this process highly complex.

Software licenses – This approach requires a client license for every VM that is protected plus the end user must purchase and install new client licenses every time a new VM is provisioned.  The management of client licenses can rapidly become challenging.

Bare metal recovery – One of the benefits of VMware is the ability to duplicate VMDK files easily to allow for the rapid creation of new VMs.  Ideally, a VMware backup would enable this functionality by providing an image-based backup and recovery feature.  Unfortunately, the traditional agent-based model does not meet this requirement.  Agent backups are file-based and so the only recovery possible is at the file level.  If a VM fails and must be recovered, then the traditional restore process of installing the OS, then the desired application and finally recovering the data will apply.

It is clear that there are benefits and challenges to the agent-based backup approach.  This model is very familiar to most backup administrators since it is nearly identical to a physical server backup process.  The approach ignores some of the advanced features in VMware such as the VMware API for Data Protection (VADP).  However, the mature application support of the agent-based model still makes it appropriate for some data types, and will maintain its relevance long into the future.

Related posts:

1. Will dedicated VMware protection solutions go the way of CDP?
2. LTO-5 and Disk-based Backup
3. CDP data protection and VMware backup: A response

A couple of weeks ago, I blogged about the benefits of Symantec’s Open Storage Technology (OST). The technology enables accelerated disk-to-disk backups (D2D) primarily over IP connections and additional value-added features. Last week, EMC responded with their announcement of BOOST for NetWorker. Insiders have told me that the BOOST architecture is essentially the same as OST although the go-to-market strategy is very different. Of course a major difference is that OST has been shipping for over 3 years and BOOST will not be available until sometime in the second half of 2010.

As discussed previously, EMC/Data Domain was unable to create a true global deduplication solution so were forced to use OST to do the heavy lifting. Ironically, they could only support Symantec NetBackup and BackupExec with the new feature because NetWorker did not offer an advanced D2D interface. The BOOST announcement addressed the issues, but raises new questions. Specifically, BOOST is positioned as an EMC only solution, and it is unclear if the API will be shared with other vendors. In my opinion, this creates a challenge for EMC/Data Domain and NetWorker. Let’s look at how the situation impacts a variety of interested parties.

End users:

Choice is critical. Customers’ needs vary widely and want the flexibility to choose the right solution for their environment. OST supports a range of choices from SMB-centric solutions to large systems targeted at enterprise datacenters. The offerings vary on numerous metrics including price, performance, scalability and reliability. SEPATON’s VTL offerings provide the fastest performance with the ability to ingest data at over 17 TB/hr and we will provide similar performance leadership with OST. If you want the highest performance D2D backups while using NetWorker, then Data Domain is your only choice. Fortunately, SEPATON’s VTL performance is maintained in NetWorker environments and so customers will have a high performance Fibre Channel option. But if you want an IP-based backup, then BOOST’s exclusively support of Data Domain is limiting. If you want choice and the ability to compare different solutions on an even playing field, you can’t have it with NetWorker. This puts the application at a competitive disadvantage versus NetBackup.

NetWorker:

BOOST brings a much needed improvement to NetWorker’s disk-based backup functionality, but unfortunately these benefits are limited to the Data Domain platform. If you compare NetWorker and NetBackup, the openness of OST is a clear advantage. Of course this is just one of many features, but it appears that NetWorker has not improved its competitive positioning in this area.

EMC:

EMC has historically been a storage centric company, and BOOST’s exclusive support of Data Domain reflects this heritage. By opening the API only to their own products, EMC is giving Data Domain a competitive advantage in IP-based backup performance. From EMC’s perspective this is good because it could drive up adoption and lock in of Data Domain. This may well be true for existing NetWorker customers; however, for customers thinking about moving to NetWorker, this “lock-in” limitation could hurt adoption.

In summary, EMC’s strategy with BOOST is all about vendor lock-in. They do not want to support third party platforms or an open architecture; instead they are forcing customers to purchase Data Domain solutions. Clearly driving more Data Domain sales is good for EMC, but the lack of choice is bad for customers. At best, BOOST maintain NetWorker’s competitive position versus NetBackup. OST’s support of multiple platforms is a compelling differentiator and competing software ISVs will compete aggressively using the mantra of openness. I believe that the lock-in strategy will ultimately hurt NetWorker marketshare and competitiveness.

Disclosure: SEPATON is an EMC NetWorker partner and we fully support NetWorker environments both with VTL and deduplication.

Related posts:

1. Data Domain & GDA – Bolt-on to the rescue
2. SEPATON Versus Data Domain
3. DeltaStor Deduplication, cont….

George Crump posted an article over on Network Computing discussing why storage is different for data protection. He makes a number of points regarding the benefits of using a storage appliance approach versus a software-only model, and for the most part, I agree with his analysis. However, there is an important point missing.

The software-only model relies on a generic software stack that can use any hardware or storage platform. This extreme flexibility also creates extreme headaches. The software provider or ISV cannot certify every hardware and environment combination and so the customer is responsible for installing, qualifying and testing their system. Initial setup can be difficult, but support can be even harder.

What happens if the product is not performing? The support complexities become difficult. Do you call your software ISV, your storage vendor, your SAN provider, your HBA vendor? There are a myriad of different hardware pieces at play and the challenge becomes how to diagnose and resolve any product issues. This is less of a problem in small environments with simple needs, and rapidly becomes an issue as data sizes grow.

The other challenge is managing change. The end user needs a solution that can adjust transparently to changing requirements such as stricter SLAs, new applications or data growth. The software-only model places the burden of system management and growth on the end user. Growing these solutions may not sound difficult, but typically involves purchasing and configuring multiple disparate pieces of hardware and software including servers, switches, storage and software licenses.  These problems get geometrically worse as systems grow in size.

Storage backup appliances simplify the situation tremendously. The vendor provides a fully qualified and supported solution with a single point of contact. If a problem arises, the appliance provider will address the issue directly thus avoiding the finger pointing issues that are common with multi-vendor solutions.  As environments change, the vendor will simplify the process and provide a supported upgrade path. However, remember that scalability varies widely and so the upgrade path will be more transparent with some solutions.

In summary, pure software-based deduplication solutions can provide strong business value in smaller homogeneous environments. However, as data sizes grow, they rapidly become complex and costly to manage.

Related posts:

1. IBM Storage Announcement
2. Data protection and natural disasters – Part 1
3. Data Protection Reporting: A Survey

The Twittersphere was abuzz yesterday with EMC’s announcement of VPLEX. For those of you who missed it, VPLEX is a storage virtualization and caching solution that presents block storage over long distances. The initial release only supports data center and metro distances with a future of continental and global reach. The announcement struck me as yet another flavor of storage virtualization which is already offered by many vendors, and got me thinking about protecting VPLEX data.

Traditional data protection architectures revolve around the concept of a master backup server supporting slave media servers and clients. The master server owns the entire backup environment and tells each server when and where to backup. The model is mature and works well in today’s datacenters where servers are static and technologies like VMotion move VM’s to new servers within the confines of the datacenter. However, the concept of global VMotion can break this model.

Now, let’s turn the discussion to VPLEX which theoretical allows you to move virtual machines freely. You can imagine a scenario where your Exchange VM is in Boston today and then is moved to San Diego tonight. That might be nice from a computing and storage standpoint, but what about backup? You have moved the system over 2500 miles and now have to protect it. Do you push the data back to Boston which could consume large amounts of bandwidth or use an alternative backup infrastructure in San Diego? If the latter, you will have to reconfigure the Exchange server backups to point to San Diego’s backup server, but what if you move it back to Boston in a week? One option is to maintain a duplicate Exchange VM in Boston, but how do you put both Exchange VM’s (Boston and San Diego) into hot backup mode simultaneously? (e.g. Instantly quiesce both servers to ensure that data is a consistent state.) Another option would be to cluster your Boston and San Diego backup servers using your WAN connection. However, this quickly becomes complicated particularly if you want the flexibility to move your Exchange server frequently to multiple locations. It rapidly becomes clear that the flexibility of VMotion combined with global data movement creates new challenges.

Virtualized infrastructures provide tremendous business value by enabling many virtual machines to run on one physical machine. Now EMC comes along and suggests that VPLEX could enhance the storage efficiency and flexibility by enabling dynamic data movement. The big problem is data protection. The process of protecting VMware has always been a challenge although VMware has made great strides with the launch of vSphere. VPLEX’s geographic movement creates new challenges. End users need a backup solution that can handle the dynamic movement of VMs over long distance while ensuring that applications are always maintained in a consistent state. The backup solution also needs robust scheduling and tracking mechanism to enable the creation, management, recovery and deletion of consistent VM backups independent of physical location.

It is unclear who will be the eventual winner in global VMware protection.  Incumbent ISVs have a strong presence, but there is room for new players with VMware-centric technology.  Clearly, EMC owns one stack with Legato, VPLEX and VMware and will try to dominate.  However, there are likely to be a range of solutions and choice is good for the end user.  The good news is that there is time since true long distance storage virtualization products will not ship until 2011.

Related posts:

1. CDP data protection and VMware backup: A response
2. Poll: VMware backup methodology
3. Agent-based VMware Backups

One of biggest challenges facing today’s datacenter managers is protecting the vast quantities of data being generated. As volumes have increased, customers have looked for larger and larger backup solutions. Multi-node global deduplication systems have become critical to enable companies to meet business requirements and EMC/Data Domain’s response to these challenges has been “add another box” which is their answer to all capacity or performance scalability questions. It appears that Data Domain has acknowledged that this argument no longer resonates and has reverted to Plan B, bolt-on GDA.

The use of the term “bolt-on” stems from a previous blog post by EMC/Data Domain’s VP of Product Management, Brian Biles. In the entry, he characterizes other deduplication vendors as bolt-on solutions, and the obvious implication is that Data Domain is better because it is not a bolt-on. Few would agree with this assertion, but it is an interesting opinion and I will return to this later.

On the surface, the GDA announcement includes all of the buzzwords including “dual controller”, “global deduplication” and “transparent load-balancing”. The solution sounds impressive, right? Well so did the DL3D 4000 and remember what happened to the DL3D family? The simple fact is that EMC/Data Domain (DD) has announced a product that may be impressive on paper, but the reality is much different. DD has been rumored to be developing a true global deduplication solution for the last few years, and it is hard to believe that this is the result.

GDA is unlike any other deduplication appliance. Every vendor (including DD pre-GDA) has built appliances that are self-contained and designed to minimize the administration, configuration and modification of backup environments. In short, the solutions are designed for simplicity. The GDA is different; it is essentially two separate DD880′s and a heavy OST client that hashes and software compresses the data and sends it to one of the two boxes based on the first digits of the hash. By forcing media servers to perform highly CPU-centric processing, the GDA moves the problem to the media server. The problem now rests on the end users shoulders to manage and size every server to ensure it meets appropriate GDA specifications. Essentially, the GDA has assimilated the media servers into its “appliance” realm and pushed processing and management activities onto the customer. It is important to note that GDA requires Symantec OST technology which is available on NetBackup and BackupExec. If you run any other backup application then GDA is not an option for you.

Here are some points to consider related to the implementation:

• How do you size the required media server upgrades to meet your performance requirements?
• Will you need to purchase new media servers to gain the processing power required?
• Who do you call if your performance does not meet expectations?
• Are you comfortable running such highly CPU intensive agents on your media servers which may already be resource constrained?
• What happens if you have some applications that backup directly to tape using a LAN free approach such as NDMP or RMAN?
• What if you are running backup applications like IBM TSM, EMC’s own NetWorker or CommVault that do not support OST?

These are real end user concerns. We have plenty of customers with 20+ media servers who would need to completely overhaul their environment to implement GDA. These customers would scoff at adding DD’s agents to their already overburdened servers.  In contrast, a true multi-node global deduplication solution like SEPATON’s can be connected and will just work as a backup target. Agents are not required when backing up to a VTL.

In summary, it appears that EMC/Data Domain is tacitly acknowledging the limitations of their hash-based architecture. They spent years trying to develop an integrated global deduplication solution and the best they could come up with is a cobbled together system including dual 880′s and an unwieldy OST client. It appears that the GDA has more in common with the DL3D4000 then I originally thought! Isn’t it ironic that the biggest opponent of bolt-on technology has so fully embraced the concept in the GDA?

Related posts:

1. SEPATON Versus Data Domain
2. NetApp and EMC Duel to the Death for Data Domain
3. NetApp and Data Domain: ‘Til death or a better offer from EMC do we part

Today SEPATON announced that we are demonstrating OST technology at Symantec vision and I created this short video demo to highlight the technology.  Enjoy!

Click here to view the embedded video.

Related posts:

1. Tradeshow giveaway gone bad: the video
2. Perspectives on Symantec OpenStorage
3. SEPATON Performance Revisited

HP recently announced the availability of LTO-5 and they are currently hosting industry luminaries at their HP Storage Day. I received a question on Twitter from John Obeto about LTO-5 and what it means to VTL and wanted to answer it here. Note that I previously blogged about LTO-5.

The challenge with data protection is ensuring that you meet your backup and recovery requirements, and most companies have fixed SLAs. The advent of LTO-5′s larger tape sizes is nice, but tape size is not the problem, the issue is real world performance. Quantum’s LTO-5 specification suggests maximum performance of 140 MB/sec which is an impressive statistic, but in practice few end users achieve this. The challenge is even greater when you think about minimum required transfer rates as discussed in my fallacy of faster tape post

Disk-based backup provides the ideal solution to address nightly backup and recovery challenges. A disk-based target will provide variable ingest speeds and can accept data as slow as the source will push it. The disk targets also allow for fast recoveries and will accelerate the process of creating LTO-5 tapes for long-term archive or DR. However, remember that the higher speeds of LTO-5 put a premium on recovery performance and so you should carefully review any disk-based solution to ensure that it can stream your LTO-5 drives.

LTO-5 is an evolutionary tape technology. It provides higher density and improved performance specifications, but the actual performance gains are less clear due to streaming challenges. LTO-5 is ideal for long-term archive or long-term DR applications where you need dense, portable and low cost storage and have infrequent access requirements. Conversely, disk-based backup is optimal for day-to-day backup operations due to the improved speed, reliability and deduplication. In summary, I believe that disk and tape are complementary backup technologies and the LTO-5 announcement further cements tape’s positioning as an ideal archive target..

Related posts:

1. Agent-based VMware Backups
2. Tale of the Tape: Musings on IBM’s 35TB Tape Announcement
3. The Fallacy of Faster Tape

The rumors of the demise of COPAN were rampant in late 2009. There was broad speculation that general operations had wound down and that the company was maintaining a skeletal staff. It was clear that COPAN’s end was near and the management team was scrambling for an exit strategy. Most people assumed that the recent silence from COPAN suggested that the company had not survived.

It was in the context of this situation that I saw a tweet last night about COPAN being acquired. The first questions were who and for how much and the tweet suggested that the answers were SGI and $2 million dollars respectively. Wow, what an amazing decline. COPAN raised $124 million dollars in multiple financing rounds and they exit the market at a $2 million valuation.

COPAN focused on MAID (massive array of idle disks). The technology allowed them to spin down unused disks to reduce the power and cooling requirements. The design included proprietary highly dense disk packaging that provided the densest storage in the industry, and actually required some datacenters to specially reinforce their flooring. They focused on $/GB and said that they offered the lowest in the industry both from an acquisition and operational cost standpoint. All of this sounded compelling from a marketing perspective, but the reality was different.

COPAN’s technology was optimized for storage density and green-ness, and not for performance. Their initial route to market was to sell a VTL appliance through an OEM agreement with FalconStor. The difficulty is that the limiting factor for many backup environments is speed. Companies must ensure that they are meeting their backup and recovery requirements and performance is typically the key element. COPAN’s positioning of “very green, but not very fast,” did not resonate. Of course, customers with specific power and cooling limitations were interested, but this represented a fraction of the market. In my opinion, this positioning error and architecture limitation limited initial adoption.

The advent of deduplication became another huge problem for COPAN. Deduplication can be highly I/O intensive and often requires high performance disk systems. In fact, some vendors even require Fibre Channel disk drives for performance. COPAN’s architecture did not align with this technology trend. They did not have the I/O performance for deduplication and their message of space, power and cooling savings was being echoed by deduplication vendors. They tried to remedy the situation by including non-COPAN disk as storage for the FalconStor deduplication landing zone, but that clearly was not a winning solution.

The other area COPAN tried to address was the archival space. They partnered with Quantum on the StorNext file system and positioned their platform as an archival target. Unfortunately they were unsuccessful in this endeavor as well.

The core problem with COPAN is that they made a bet on a technology that did not address a primary customer market pain. End users appreciated the power and cooling benefits of the solution, but that need was secondary to the real requirement of backup and recovery performance. Deduplication technology further eroded their position by bringing substantial power and cooling benefits to traditional disk platforms. COPAN was stuck with a proprietary MAID technology with a limited market opportunity. The result is that COPAN’s technology became irrelevant due to deduplication and more recently disk spin-down technologies, and this was reflected in the company’s dwindling financial performance. SGI is now the owner of COPAN’s technology and it will be interesting to see how they bring it to market.

Lessons learned

The key lesson is that success depends on addressing real customer pains. COPAN tried to build a business on a feature, power and cooling efficiency, while customers wanted a solution to their real backup and recovery pains. This disconnect resulted in a solution that was not well aligned with the market and hindered COPAN’s success. If you are an emerging vendor, you must thoroughly understand the markets that you address and focus your product on solving customer pains. Anything less will inevitably lead to failure.

Related posts:

1. Lessons from the Sidekick debacle