About Restore » Virtual Tape

The Demise of the NearStore VTL: A historical perspective

Jay Livens — Wed, 03 Feb 2010 17:57:34 +0000

Rumors have been circulating for months about the demise of NetApp’s VTL offering. Today, Beth Pariseau from SearchDataBackup published the first public confirmation that development on the product has ceased. It is not a surprise, but makes for an interesting case study.

NetApp acquired VTL technology with their purchase of Alacritus for $11 million back in 2005. Alacritus provided a software only VTL solution that ran on a Linux platform. Their product specifications appeared impressive, but they had limited success in the US. Our partners in Asia saw them more frequently. For NetApp, the acquisition made sense because it represented a relatively cost-effective entry into the rapidly growing VTL market. However, as in most things, the difficulties were in the details.

NetApp’s core intellectual property is their ONTAP operating system and associated WAFL filesystem. These components provide the intelligence and value-added features of their arrays. The challenge for NetApp after acquiring Alacritus was the integration of the two technologies.

NetApp’s short term strategy after the purchase was to build a VTL using Linux, Alacritus, x86 servers and existing disk shelves, and they released their first system in 2006. This leveraged their hardware design and configuration, but bypassed their core software IP. Customers could reuse VTL disk shelves in their filers if they decommissioned the VTL, but there was no ability to share the same storage between filers and VTLs. This is your classic island of storage problem.

The next trend in the market was deduplication. NetApp recognized the importance of the technology and announced A-SIS deduplication for their filers in May of 2007. However, the Alacritus VTL did not support ONTAP/WAFL and so the technology could not be reused. NetApp then embarked on a project to develop a second deduplication engine (after A-SIS) for use on their VTL. They said that the new deduplication system would be available in June of the same year, and I believe that they underestimated the difficulty of creating a new engine. The development took far longer than expected and they announced the product in October of 2008. To this day, we have seen few implementations of NetApp VTLs with deduplication, and I often wonder if this is because their new deduplication engine does not work effectively. Given the limited success, it is not surprising that NetApp has officially ceased development.

In my opinion, NetApp’s VTL failed due to a lack of commitment. To be successful, they needed to port the Alacritus solution to ONTAP/WAFL to allow it to co-exist transparently with existing technology. I believe that NetApp was focused on snapshots and replication for backup and recovery and so the NearStore VTL was not given the attention it deserved. Had they integrated Alacritus and ONTAP, they had the potential to create a unified filer, VTL and deduplication appliance which would have been very competitive. Instead, they chose to leave Alacritus as a separate product, and NetApp’s aggressive pursuit of Data Domain marked the end of the NearStore VTL.

The irony is that I believe NetApp had an opportunity to be an early player in the VTL space. They had the relationships and footprint to aggressively sell the solutions. Unfortunately, their critical missteps and lack of focus limited their success.

Streaming LTO-5

Jay Livens — Fri, 14 Aug 2009 18:17:46 +0000

Chris Mellor (twitter:@Chris_Mellor) recently posted an article over at The Register about LTO-5 entitled Is LTO-5 the last harrah for tape?. He makes an interesting point about the future of LTO and whether LTO-5 will be the last generation of the technology. Most of the comments on the article disagree with Chris’s opinion.

I believe that there is another major issue with LTO-5 that must be addressed. The challenge with LTO (and most other tape technologies) is its limited ability to throttle performance. Users must carefully manage their environment to ensure that they stream their drives or else backup performance will decline dramatically. As drives become faster, the challenge of optimizing your environment for the technology becomes more difficult. You can read more about this in my blog post entitled The Fallacy of Faster Tape.

Backup software ISVs have recognized physical tape’s limitations and have developed technologies to improve backup speeds; a classic example is multiplexing. Customers can multiplex their backups to improve ingest performance; however, the result is a degradation in restore speed. The key takeaway is that faster drives bring new infrastructure challenges that must be addressed. To illustrate this point numerically, I updated the chart from The Fallacy of Faster Tape with estimated LTO-5 performance.

Click for larger image

The chart shows that the minimum usable performance for LTO-5 is between 62 and 90 MB/sec and the required streaming speed increases linearly with the number of drives. The streaming requirement is the crux of the issue and creates a major management and infrastructure challenge.

The best solution to this problem is to introduce disk as a backup target. Disk provides infinitely variable ingest speeds and will not suffer the shoe-shining penalties of tape. However, if tape copies are required, you must carefully review the restore performance of your disk device because shoe-shining is still a possibility if the disk system cannot restore data faster than 60 MB/sec. This issue is particularly relevant in the context of deduplication since the processing of rebuilding the deduplicated data can impact restore performance. I wrote about this problem in another blog post. SEPATON has architected the S2100-ES2 and DeltaStor to provide the fastest backup and recovery performance to address these requirements.

In summary, LTO-5 shows the LTO consortium’s commitment to increase tape density and performance. The real world performance improvements experience by end users transitioning to LTO-5 will vary widely. This has always been the case with tape technology, but the added streaming requirements of LTO-5 increases the challenge. However, physical tape still has a place in many enterprises as a deep archival medium and LTO-5 is well suited for this role.

When is a node not a node?

Jay Livens — Thu, 30 Jul 2009 19:52:19 +0000

One of the things that irks me is when press/analysts/vendors compare a competitor’s solution to a one node SEPATON solution. SEPATON’s VTL as well as our DeltaStor deduplication and DeltaRemote replication products rely on our DeltaScale™ architecture which is designed around the concept of grid scalability. The grid allows us to scale dynamically and transparently across multiple independent nodes. This is very different from competing solutions that rely on a monolithic server approach.

Creating a truly dynamic and grid-based VTL with deduplication and replication is difficult. That’s why most vendors have opted for a single node approach. In order to achieve performance, they include the largest servers feasible to try to minimize the server hardware bottleneck. These are typically 2U+ servers that are loaded with huge amounts of RAM, four or more of the fastest possible CPUs and often SSD’s to hold the deduplication index. The challenge with this approach is that the customer must buy the right long-term solution to meet their needs on day one. If they do not, they will either have to perform a forklift upgrade to a new product version with a faster server or purchase multiple iterations of the same system—and cobble them together.

SEPATON’s DeltaScale architecture relies on a grid of relatively inexpensive 1U nodes. All of these nodes appear as one system image and share common back end storage, virtual libraries/drives and a deduplication repository. The benefit of this approach is that a customer is not forced to buy the maximum performance required on day one. Instead,they can buy what they need today and increase performance later by adding nodes to the grid. The system automatically recognizes the increased performance and integrates it, load balancing all processing across all nodes without operator intervention. This also helps with the economics of the system since SEPATON’s nodes are less expensive than those of the competition. SEPATON could easily inflate single-node performance by providing similarly powerful nodes as the competition does, but we believe that the benefit of low cost and granular performance in our current design is a better fit with our customers’ needs.

As a further clarification, I thought that it would be helpful to give an idea of the cost differential between a SEPATON node and a monolithic one from a competitor. In this case, I assume a node such as that recently announced by Data Domain. I configured systems on Dell’s website and the price stated is based on estimated configurations, and excludes any custom parts or volume discounts. The point is to illustrate the order of magnitude difference. Click on the price to view a PDF of the server quote from Dell.

	SEPATON Node	Competitor Node
Dell list price	$6,810	$25,917

The result is that a SEPATON node is about one quarter the price of those used in competing systems.

In summary, SEPATON’s DeltaScale architecture supports a grid of thin nodes while competing solutions typically rely on one over sized node. The SEPATON approach provides the ability to scale performance dynamically in a single system image through the addition of nodes and the alternative is to increase performance by adding independent systems.

War Stories: Diligent

Jay Livens — Fri, 01 May 2009 20:16:29 +0000

As I have posted before, IBM/Diligent requires Fibre Channel drives due to the highly I/O intensive nature of their deduplication algorithm. I recently came across a situation that provides an interesting lesson and an important data point for anyone considering IBM/Diligent technology.

A customer was backing up about 25 TB nightly and was searching for a deduplication solution. Most vendors, including IBM/Diligent, initially specified systems in the 40 – 80 TB range using SATA disk drives.

Initial pricing from all vendors was around $500k. However as discussions continued and final performance and capacity metrics were defined, the IBM/Diligent configuration changed dramatically. The system went from 64TB to 400TB resulting in a price increase of over 2x and capacity increase of 6x. The added disk capacity was not due to increased storage requirements (none of the other vendors had changed their configs) but was due to performance requirements. In short, they could not deliver the required performance with 64TB of SATA disk and were forced to include more.

The key takeaway is that if considering IBM/Diligent you must be cognizant of disk configuration. The I/O intensive nature of ProtectTier means that it is highly sensitive to disk technology and so Fibre Channel drives are the standard requirement for Diligent solutions. End users should always request Fibre Channel disk systems for the best performance and SATA configurations must be scrutinized. Appliance-based solutions can help avoid this situation by providing known disk solutions and performance guarantees.

SEPATON Performance — Again

Jay Livens — Wed, 25 Feb 2009 23:20:39 +0000

Scott from EMC has challenged SEPATON’s advertised performance for backup, deduplication, and restore. As industry analyst, W. Curtis Preston so succinctly put it, “do you really want to start a ‘we have better performance than you’ blog war with one of the products that has clustered dedupe?” However, I wanted to clarify the situation in this post.

Let me answer the questions specifically:

1. The performance data you refer to with the link in his post three words in is both four months old, and actually no data at all.

SEPATON customers want to know how much data they can backup and deduplicate in a given day. That is what is important in a real life usage of the product. The answer is 25 TB per day per node. If a customer has five nodes and a twenty-four hour day, that’s 125 TB of data backed up and deduplicated. This information has been true and accurate for four months and is still true today.

2. Sepaton claims the same performance for deduplication on and off. Which is not realistic. They make no mention of what happens when you do both tasks (ingest and deduplication) simultaneously.

Our numbers are conservative and include ingest, deduplication and house-cleaning tasks. Backup is even faster than that. We use the 25TB per node per day as our metric so our customers get the bottom line information they need to make a sound business decision about how fast we can get their data backed up and deduplicated. They don’t care if it backs up slightly faster than that.

As for dedupe off—a SEPATON VTL with 16 nodes can backup at 34.5 TB per HOUR.

3. Just like TSM deduplication. Coming soon.

TSM deduplication is GA.

4. They do have an issue restoring from deduplicated data, they just don’t want to discuss it.

Not sure where you get this information. We use a process called forward referencing so our restore speed from deduplicated data is the fastest in the industry. We store the most current data in full and deduplicate older data with pointers forward in time to it. The processing load of restoring day-old or weeks-old data is miniscule. Think about it, the newer the data, the less reconstitution necessary. Just the opposite of the EMC solution. In fact, data would have to be ready for cold storage (six months) before it was old enough to slow a SEPATON VTL down to EMC’s restore speed. Even then, it would likely be faster.

5. We have the ability to size how big the cached data pool is.

You highlight that EMC’s restore performance from the deduplication repository is a whopping 75% slower than ingest. The DL3D has the ability to size the cached data pool because it needs a cached data pool. SEPATON doesn’t. We restore as fast as we back up. Period. If the data is months old, we restore it nearly as fast as we backed it up. The slow down is smaller than a rounding error.

Customer perspectives on SEPATON, IBM and Data Domain

Jay Livens — Wed, 14 Jan 2009 16:47:29 +0000

SEPATON issued a press release on Monday that is worth mentioning here on the blog. SearchStorage also published a related article here. The release highlights MultiCare a SEPATON customer that uses DeltaStor deduplication software in a two-node VTL.

In the release, the customer characterizes their testing of solutions from Diligent/IBM (now IBM TS7650G) and Data Domain. Specifically, they mention that the TS7650G was difficult to configure and get running and that the gateway head nature of the product also made it difficult for them to scale capacity. These difficulties illustrate the challenges of implementing the TS7650G’s head only design. With this solution, the burden of integrating and managing the deduplication software and disk subsystem falls on the end user. Contrast this with a SEPATON appliance that manages the entire device in a fully integrated, completely automated fashion.

They had a typical Data Domain experience. That is, their initial purchase looked simple and cost effective but rapidly become complex and costly. In this case, MultiCare hit the Data Domain scalability wall, requiring them to purchase multiple separate units. The result is that MultiCare had to perform two costly upgrades and had to rip and replace their Data Domain solutions with newer, faster units. Scalability is the challenge with Data Domain solutions and it is not uncommon for customers to purchase one unit to meet their initial needs and then be forced to add additional units or perform a forklift upgrade.

As MultiCare found, customers must thoroughly understand their requirements when considering deduplication solutions. They tested the head-only approach and found it to be too complex to operate and manage to meet their needs. They tried the small appliance approach and found that they outgrew their initial system and were forced to pursue costly upgrades. In the end, they recognized that the best solution for their environment was a highly scalable S2100-ES2 solution which provided the performance and scalability that could not be achieved with either the TS7650G or Data Domain.

Falconstor, SIR and OEMs

Jay Livens — Fri, 05 Dec 2008 20:49:04 +0000

This article on Byteandswitch.com highlights enhancements to FalconStor’s SIR deduplication platform, but I have to wonder whether anyone cares. FalconStor was a big player in providing VTL software to OEMs; but their deduplication software has been largely ignored.

FalconStor had their heyday in VTL. They aggressively pursued OEM deals with large vendors including EMC, IBM, and Sun. EMC was the most successful with their EDL family of products. As the market moved to deduplication, you would think that FalconStor would be the default OEM supplier of deduplication software as well. You would be wrong.

Ironically, FalconStor’s VTL success was their downfall in deduplication. Their OEMs realized that they were all selling the same VTL software and did not want to repeat the situation with deduplication. EMC and IBM, have already announced that they are using alternative deduplication providers.

EMC (26% of FalconStor revenue in 2007) partnered with Quantum. They now sell VTL appliances based exclusively on Quantum deduplication technology, the Rube Goldberg DL3D 4000 for those still wanting to use the FalconStor with Quantum deduplication, and the non-deduplication enabled EDL. EMC has the challenging task of differentiating each solution and supporting the disparate products. FalconStor will be impacted by EMC’s partnership, and if EMC’s DL3D solutions take off, the impact could be substantial.

IBM decided to take the M&A route and purchased Diligent. Unlike EMC, who at least tried to create a migration path to deduplication with the DL3D 4000, there is no option with IBM. Existing TS75xx VTLs will have to be forklift upgraded to the new TS7650G which is a complex and costly process. This defection by another OEM is another blow to FalconStor and SIR.

In summary, two out of the three major FalconStor OEMs are looking elsewhere for deduplication. The third OEM, Sun, announced a partnership with Diligent in 2007 although it appears that Sun has now focused on FalconStor exclusively. It still remains unclear who is actually selling or buying this stuff. Clearly not IBM or EMC and we rarely see Sun. I have not had a request for a competitive update on FalconStor or Sun for over a year. We compete with almost everyone at different times and it is very odd that SIR is never mentioned. It makes you wonder whether there is something else going on here causing the OEM defections.

In short, it appears that FalconStor’s position in OEM marketings is eroding and ironically, this is attributable to their previous success in VTL software. FalconStor faces a difficult task of rebuilding their OEM business in a down economy.

Choosing a Data Protection Solution in a Down Economy

Jay Livens — Wed, 19 Nov 2008 21:59:23 +0000

I hate to turn on the TV these days because it is full of bad news. There always seems to be some pundit talking about troubles in the housing market, credit markets, automotive industry, consumer confidence and so many other areas. It does not take a rocket scientist to recognize that the economy is in tough shape right now. As a reader of this blog, you are likely feeling some of the pain in your budget. This obviously brings up an important question: how do I justify IT purchases in these environments.

In situations like these, IT departments must go back to the basics. Purchases must be all about ROI. You must look beyond just acquisition cost and consider how a given solution can save your organization money both upon acquisition and into the future.

The interesting thing about ROI is that there are many components that should be considered. Look at both capital expenditures and operational expenditures over time. This reveals an interesting irony. Tape may have a low acquisition cost, but a much higher on an operating cost. Tape will also cost you in annual capex when need to scale it. You may save money when purchasing it and yet find it very expensive to operate over the long run.

When thinking about costs, some of the key metrics you should consider include:

Cost of system management
Consider the inherent complexity and manual labor required to manage a given system. If one system meets your needs today, what will your environment look like in one, two or five years and what will be the resulting impact on your operational costs? There is an added cost for managing and implementing multiple systems versus a single system; you need to decide on the amount. A single highly scalable solution can help with this cost.

Cost of data center space
The high cost of data center space may make adding systems cost prohibitive. You need to get the best utilization out of existing space and must consider system footprint. Clearly deduplication can help here and the more you can reduce your capacity requirements, the better off you are. Server virtualization can also help since you can have multiple virtual servers in the footprint of one physical server.

Power and cooling cost
As the prices of natural resources have increased so has the price of electricity. You need to find ways to leverage your existing electrical and cooling infrastructure more efficiently. Deduplication can help with this since you can use it to reduce your system footprints. Server virtualization can also help by reducing the number of servers and related power and cooling requirements.

Maintenance cost
Many vendors include this in the purchase price. However, at some point, you will need to renew maintenance. Beware of vendors that charge dramatically higher rates for maintenance renewals and uses this as leverage to sell a new system. In other cases, technology becomes so obsolete that it is unsupportable. In these cases, you need to think about a solution that provides backward compatibility so as new product features and components come out, they can be integrated into the existing system without a forklift upgrade. (e.g. transparently move from 2 Gb to 4 Gb FC, non-dupe to dedupe enabled or from one drive size to another..)

All of these metrics must be considered when building an ROI on the solution. Compelling ROI data will allow you to build a case for upgrading your infrastructure that is financially justified. Prior to today’s difficult times, ROI justification was important, but since the economic difficulties, it has become vital.

Surviving A Down Economy – A vendor Perspective

NetApp Dedupe: The Worst of Inline and Post-process Deduplication

Jay Livens — Thu, 30 Oct 2008 18:22:29 +0000

NetApp finally entered the world of deduplication in data protection. While they have supported a flavor of the technology in their filers since May 2007, they have never launched the technology for their VTL. Why? Because their VTL does not use any of the core filer IP. It relies on an entirely separate software architecture that they acquired from Alacritus. Thus all the features of ONTAP do not apply to their VTL. However, I digress from the topic at hand.

I posted recently about three different approaches to deduplication timing: inline, post process and concurrent process. I talked about the benefits of each and highlighted the fact that post process and concurrent process benefit from the fastest backup performance since deduplication occurs outside of the primary data path while inline benefits from the smallest possible disk space since undeduplicated data is never written to disk. Now comes NetApp with a whole new take. Their model combines the worst of post process and inline, by requiring a disk holding area and reduced backup performance. After all this time developing the product, this is what they come up with? Hmmm, maybe they should stick to filers.

Below is a screenshot from NetApp’s web site where they specify performance with deduplication on and off.

Looking at the numbers you would think that they are deduplicating data inline since performance decreases by about 50% when deduplication is enabled. You would be wrong; according to The Register, they are post processing (albeit with some deduplication work upfront). The article indicates that they generate “rolling hashes” in real time while still storing the data in its entirety for future deduplication.

This is the worst possible combination. They are doing the CPU intensive part of deduplication inline thus reducing performance, and are not gaining any immediate disk space savings! Additionally, since this is post-process, the actual removal of redundant data will take place at a later time and will require substantial I/O resources and will likely slow backup and restore operations. NetApp does not disclose the performance impact of the actual deduplication process, but it is a single node system and so performance is not likely to exceed 400 MB/sec.

One final addition is that NetApp continues to use RAID 5 in this platform. This is dangerous since their deduplication engine will replace data with pointers which can live anywhere on the disk system. A double disk fault on any shelf would likely result in catastrophic data loss. Most other deduplication vendors including SEPATON utilize RAID 6 to further protect data. EMC blogger Scott Waterhouse has posted on this topic on numerous occasions and I defer to his analysis on the issue. Fundamentally, this is a major flaw in NetApp’s VTLs and it is ironic that NetApp in their whitepapers recommend against using RAID 5 on their filers and yet offer no alternative on the VTL.

Data protection and natural disasters – Part 2

Jay Livens — Wed, 24 Sep 2008 15:02:17 +0000

In part 1, I touched on four of the most common challenges with data restoration in a disaster scenario. In this post, I will review some other key considerations. These examples focus on the infrastructure required after a disaster has occurred.

Available restore target
After a disaster occurs, the immediate response is to try to get critical systems up and running. The process involves restoring data to servers that are in place at the DR site. The challenge is ensuring that you have enough computing resources available. What good is a restore operation if you have nothing to restore the data to? This problem gets worse in environments where new servers are frequently provisioned in the primary datacenter.

The biggest challenge is that IT budgets are limited and companies often cannot have a matching DR server for every production server. The good news is that server virtualization like VMware can help. With VMware you can provision multiple virtual servers on one physical server, thus reducing your physical server requirements at the remote site. The downside is that performance may suffer on some virtual servers due to resource contention; however, in most instances, full recovery with a performance impact is better than no recovery. The best way to ensure that the DR site is adequately provisioned is to perform periodic tests where you bring up all of your DR servers and ensure that they are in working order and that adequate computing resources are available.

Restore Priority
After a disaster occurs, the administrator must decide what systems to recover first. This can be a complex decision given the needs of various departments and limited availability of tape restoration resources. Trying to create these priorities after a disaster has occurred is a recipe for trouble. Administrators must focus on restoring data and not negotiating restore priorities with various stakeholders.

The best solution here is to understand the criticality of various servers BEFORE the disaster occurs. You need to understand your environment and categorize systems based on the level of criticality for your organization. For example, the corporate order entry system is probably more important than each employee’s fileshare. Once you have your priorities clearly defined, you can focus on restoring the most critical data after a disaster occurs.

Environmental
After the disaster occurs, you will immediately need to provision computing resources which will be used as targets for recovery operations. The challenge is ensuring that you have a sufficient infrastructure for the new resources. Power, cooling and computing power is vital and you must ensure that you have enough of these to meet your requirements. This can be especially troublesome if multiple entities at the DR site are all trying to simultaneously implement their plans

This problem is avoidable with planning and testing. The best solution is to perform periodic DR tests to ensure that you have enough of these limited resources. Additionally, you must ensure that you have the appropriate SLAs in place with your DR provider to guarantee that they can provide enough power and cooling resources to meet your requirements.

Human Resources
In order for the DR process to begin, you need people to provision the DR environment and begin the restore process. You can use existing team members to handle the process; however, in cases of a major disaster, those people may be unable to reach the DR site and be unavailable by telephone. This can be very problematic and brings an interesting debate about whether it is better to have a relatively close (easy to access, but more at risk) or distant (difficult to access, but safer) DR site.

There is no simple solution to the above problem. As mentioned, one solution could be to choose a closer DR site, but that brings other issues. This decision is best left to each company to decide based on their business requirements. At the very least, companies should pro-actively monitor situations and be ready to send at least one human resource to a remote site in advance of any recognizable DR scenario. Of course, in some cases, there is no advanced warning and in those cases you should ensure that you have IT experts geographically disbursed as part of your business operations.

All of the above challenges are addressable with careful planning. One of the most important activities a team can perform is periodic DR testing. These simulations allow companies to refine their DR processes and minimize the downtime if a disaster occurs. All of these activities should happen in the context of a larger DR plan which should include elements of testing, human resources, application prioritization and other elements.