About Restore » D2D

Boost vendor lock-in

Jay Livens — Wed, 19 May 2010 15:05:15 +0000

A couple of weeks ago, I blogged about the benefits of Symantec’s Open Storage Technology (OST). The technology enables accelerated disk-to-disk backups (D2D) primarily over IP connections and additional value-added features. Last week, EMC responded with their announcement of BOOST for NetWorker. Insiders have told me that the BOOST architecture is essentially the same as OST although the go-to-market strategy is very different. Of course a major difference is that OST has been shipping for over 3 years and BOOST will not be available until sometime in the second half of 2010.

As discussed previously, EMC/Data Domain was unable to create a true global deduplication solution so were forced to use OST to do the heavy lifting. Ironically, they could only support Symantec NetBackup and BackupExec with the new feature because NetWorker did not offer an advanced D2D interface. The BOOST announcement addressed the issues, but raises new questions. Specifically, BOOST is positioned as an EMC only solution, and it is unclear if the API will be shared with other vendors. In my opinion, this creates a challenge for EMC/Data Domain and NetWorker. Let’s look at how the situation impacts a variety of interested parties.

End users:

Choice is critical. Customers’ needs vary widely and want the flexibility to choose the right solution for their environment. OST supports a range of choices from SMB-centric solutions to large systems targeted at enterprise datacenters. The offerings vary on numerous metrics including price, performance, scalability and reliability. SEPATON’s VTL offerings provide the fastest performance with the ability to ingest data at over 17 TB/hr and we will provide similar performance leadership with OST. If you want the highest performance D2D backups while using NetWorker, then Data Domain is your only choice. Fortunately, SEPATON’s VTL performance is maintained in NetWorker environments and so customers will have a high performance Fibre Channel option. But if you want an IP-based backup, then BOOST’s exclusively support of Data Domain is limiting. If you want choice and the ability to compare different solutions on an even playing field, you can’t have it with NetWorker. This puts the application at a competitive disadvantage versus NetBackup.

NetWorker:

BOOST brings a much needed improvement to NetWorker’s disk-based backup functionality, but unfortunately these benefits are limited to the Data Domain platform. If you compare NetWorker and NetBackup, the openness of OST is a clear advantage. Of course this is just one of many features, but it appears that NetWorker has not improved its competitive positioning in this area.

EMC:

EMC has historically been a storage centric company, and BOOST’s exclusive support of Data Domain reflects this heritage. By opening the API only to their own products, EMC is giving Data Domain a competitive advantage in IP-based backup performance. From EMC’s perspective this is good because it could drive up adoption and lock in of Data Domain. This may well be true for existing NetWorker customers; however, for customers thinking about moving to NetWorker, this “lock-in” limitation could hurt adoption.

In summary, EMC’s strategy with BOOST is all about vendor lock-in. They do not want to support third party platforms or an open architecture; instead they are forcing customers to purchase Data Domain solutions. Clearly driving more Data Domain sales is good for EMC, but the lack of choice is bad for customers. At best, BOOST maintain NetWorker’s competitive position versus NetBackup. OST’s support of multiple platforms is a compelling differentiator and competing software ISVs will compete aggressively using the mantra of openness. I believe that the lock-in strategy will ultimately hurt NetWorker marketshare and competitiveness.

Disclosure: SEPATON is an EMC NetWorker partner and we fully support NetWorker environments both with VTL and deduplication.

Streaming LTO-5

Jay Livens — Fri, 14 Aug 2009 18:17:46 +0000

Chris Mellor (twitter:@Chris_Mellor) recently posted an article over at The Register about LTO-5 entitled Is LTO-5 the last harrah for tape?. He makes an interesting point about the future of LTO and whether LTO-5 will be the last generation of the technology. Most of the comments on the article disagree with Chris’s opinion.

I believe that there is another major issue with LTO-5 that must be addressed. The challenge with LTO (and most other tape technologies) is its limited ability to throttle performance. Users must carefully manage their environment to ensure that they stream their drives or else backup performance will decline dramatically. As drives become faster, the challenge of optimizing your environment for the technology becomes more difficult. You can read more about this in my blog post entitled The Fallacy of Faster Tape.

Backup software ISVs have recognized physical tape’s limitations and have developed technologies to improve backup speeds; a classic example is multiplexing. Customers can multiplex their backups to improve ingest performance; however, the result is a degradation in restore speed. The key takeaway is that faster drives bring new infrastructure challenges that must be addressed. To illustrate this point numerically, I updated the chart from The Fallacy of Faster Tape with estimated LTO-5 performance.

Click for larger image

The chart shows that the minimum usable performance for LTO-5 is between 62 and 90 MB/sec and the required streaming speed increases linearly with the number of drives. The streaming requirement is the crux of the issue and creates a major management and infrastructure challenge.

The best solution to this problem is to introduce disk as a backup target. Disk provides infinitely variable ingest speeds and will not suffer the shoe-shining penalties of tape. However, if tape copies are required, you must carefully review the restore performance of your disk device because shoe-shining is still a possibility if the disk system cannot restore data faster than 60 MB/sec. This issue is particularly relevant in the context of deduplication since the processing of rebuilding the deduplicated data can impact restore performance. I wrote about this problem in another blog post. SEPATON has architected the S2100-ES2 and DeltaStor to provide the fastest backup and recovery performance to address these requirements.

In summary, LTO-5 shows the LTO consortium’s commitment to increase tape density and performance. The real world performance improvements experience by end users transitioning to LTO-5 will vary widely. This has always been the case with tape technology, but the added streaming requirements of LTO-5 increases the challenge. However, physical tape still has a place in many enterprises as a deep archival medium and LTO-5 is well suited for this role.

The Fallacy of Faster Tape

Jay Livens — Thu, 06 Nov 2008 14:10:54 +0000

I often talk about disk-based backup and virtual tape libraries (VTL) and wanted to discuss physical tape. While VTLs are popular these days, tape is still in widespread use. LTO tape, the market share leader, continues to highlight increased density and performance. Do not be fooled with these claims. In the real world faster tape often provides little or no improvement in backup and/or restore performance. Ironically, faster tape increases (not decreases) the need for high performance disk devices like VTLs. Let me explain.

Modern tape drives use a linear technology where the tape head is stationary and the tape moves at high speed above it. Through each generation of LTO, the tape speed is largely unchanged while tape density doubles. At the same time, LTO drives have not expanded their ability to vary the speed of tape. Thus if you go from LTO-3 to LTO-4, you have doubled the density of your tape and you must double the throughput of data handled by the drive to keep tape speed unchanged. Why does tape speed matter? Because LTO tapes have a limited ability to throttle tape speeds, your performance will suffer terribly if you cannot meet the drives minimum streaming requirement.

If you are unable to stream enough data to your tape drives as mentioned above, the tape drive will go into a condition called “shoe shining” where it is constantly stopping and starting. It will try to stop when its buffer empties, but the tape is moving so fast that it will overshoot its stopping point and need to slowly stop, rewind to where it stopped writing and begin writing again. The tape moves forward and backward like shoe shine cloth. This process causes a massive reduction in performance and excessive wear on the tape drives and media. The table below comes from a Quantum whitepaper entitled “When to Choose LTO-3″ and highlights the real world performance requirements of LTO-2 and LTO-3. I have estimated LTO-4 requirements for completeness.

Click for larger image

Note how the minimum usable performance approximately doubles with each tape generation. As you upgrade tape technology, the challenge of streaming becomes greater and greater. Clearly, if you can stream LTO-3 there is no guarantee that you will be able to stream LTO-4. The ultimate irony is that upgrading to the latest tape technology often reduces not improves backup performance.

The only way to gain the benefit of the new technology is to thoroughly review and perhaps even re-architect your backup environment for the new drives. This creates a substantial hidden cost. Bringing in a disk-based backup target can often simplify the upgrade process.

Disk-based backup provide tremendous benefits in tape environments. They act as a variable speed backup target and will accept data at any rate you can send it. This frees you from the challenge in the tape world of maintaining a minimum throughput. Additionally, a VTL solution like the SEPATON S2100-ES2 provides very fast performance and can stream to LTO-4 drives and eliminate shoe shining if tape copies are required. You can often reduce the re-architecture costs associated with a tape upgrade by including a disk-based backup solution. Additionally, you can also reduce the amount of tape hardware required since the VTL will become the primary backup target of your nightly backups and you only need enough tape hardware to complete the tape copy process, if required, within your window.

InformationWeek on NEC HYDRAstor

Jay Livens — Fri, 12 Sep 2008 12:47:05 +0000

Howard Marks recently posted an interesting article about NEC’s HYDRAstor over on his blog at InformationWeek. He discusses the product and how the device is targeted at backup and archiving applications. He makes some interesting points and mentions SEPATON. I wanted to respond to some of the points he raised.

…[the system starts with] a 1-accelerator node – 2-storage node system at $180,000…

Wow, this is an expensive proposition. Comparing this to the list price of a SEPATON S2100-ES2, you could get 2x the storage and 2x the performance for about the same price. Alternatively, you could purchase a configuration with equivalent capacity and performance for about 50% less.

….[the largest system is] a 110-TB 16.5-GB* [sic] behemoth that is like Commodore Vanderbilt’s yacht — “if you have to ask how much it costs you can’t afford it….

* This must be a typo. I believe that the author is actually saying 16.5 GB/hr which matches NEC’s website.

He is right that the cost would be prohibitive for a solution of this size. At a minimum the solution requires 55 accelerator nodes (15,600/300) which results in a list price of $2.75 million, but of course storage is required as well. If we assume one storage node per accelerator, the price jumps to $6.6 million. Now I understand his Vanderbilt quip, but their prices are way out of the industry norm.

Finally, creating a system to validate that information is cost prohibitive. Have they purchased 55 accelerator nodes and 55 storage nodes and validated this performance and configuration? My guess is not. SEPATON’s architecture is designed to support 32 nodes (19,200 MB/sec performance, faster than HYDRAstor), but we do not promote that configuration since we have not qualified it yet. I guess if we wanted to be equally as aggressive, we could claim that performance, but instead we decided to be more conservative and stay with 16 nodes (9,600 MB/sec).

A Hydrastor grid’s accelerator nodes manage the cluster’s file system, allowing backup and archiving applications access via a CIFS or NFS / NAS like interface

NFS/CIFS are well established and widely used in the storage industry; however, in the data protection space, NFS/CIFS is primarily used only in small implementations. VTL is the common interface in large environments due to its simplicity to implement and mature support in backup applications. I am not saying that a massively scalable filer is a bad thing, but rather that data protection is not an ideal market for the technology.

The new accelerator nodes are available with two 10-Gb Ethernet ports eliminating the 1-Gb Ethernet (GigE) bottleneck that’s held NAS backup appliances behind VTLs in backup performance.

A HYDRAstor accelerator has two x 10 Gb Ethernet ports which equates to about 2000 MB/sec, and yet they can only achieve 300 MB/sec throughput? Additionally, in order for the customer to even see that, they will need to implement costly 10 GigE. This makes for tough economics because you not only have to purchase the system but also a new network infrastructure. SEPATON achieves better performance, 600 MB/sec per node, with a substantially cheaper interconnect, 4 Gb FC. (2 front end and 2 back end ports per node, up to 16 nodes).

In the second point, the author indicates that GigE has limited NAS performance. It certainly is true that one GigE port is much slower than 2 Gb or 4 Gb Fibre Channel, but many vendors have solved this by adding multiple GigE ports to their platforms. The other issue is that CIFS/NFS requires more computing overhead on the backup server which results in slower single stream performance. For example, let’s say you have a very powerful 4 TB Oracle server that you are backing up to a 300 MB/sec CIFS/NFS and a 300 MB/sec VTL. Fibre Channel is a much more efficient protocol which means that the backup will run close to the 300 MB/sec. In contrast, the NFS/CIFS backup will run at a fraction of this speed due to the overhead in the system and this bottleneck cannot be resolved by simply adding a faster Ethernet pipe.

In summary, HYDRAstor certainly has interesting technology. The reality of today’s environment is that business value and ROI are key. Certainly, NEC can promote a bleeding edge grid solution, but it comes down to the ROI for customers and in that area, SEPATON has an advantage versus HYDRAstor.

Analyst Commentary on VTL

Jay Livens — Thu, 28 Aug 2008 14:34:49 +0000

I am often perusing industry related sites to find what people are saying about disaster recovery and data protection. Most of these sites rely on independent contributors to provide the content. Given the myriad of viewpoints and experience levels, it is not uncommon to see a wide range of commentaries, some consistent with industry trends, and others not. I keep this in mind when reading these articles and generally ignore inconsistencies; however once in a while an article is so egregiously wrong that I feel a response is necessary.

In this case, I am referring to an article appearing in eWeek where the author makes gross generalizations about VTL that are misleading at best. Let’s walk through his key points:

VTLs are complex

I completely disagree. The reason most people purchase VTLs is that they simplify data protection and can be implemented with almost no change in tape policies or procedures. This means that companies do not have relearn new procedures after implementing a VTL and thus the implementation is relatively simple and not complex like he suggests.

He also mentions that most VTLs use separate VTL software and storage. This is true for solutions from some of the big storage vendors, but is not the case with the SEPATON S2100-ES2. We manage the entire appliance including storage provisioning and performance management.

Finally, he complains about the complexity of configuring Fibre Channel (FC). While it is true that FC can be more complex than Ethernet it really depends on how you configure the system. One option is to direct connect the VTL which requires none of the FC complexities he harps on. He also glosses over the fact that FC is much faster than the alternatives which is an important benefit. (My guess is that he is comparing the VTL to Ethernet, but he never clearly states this.)

VTLs increase total backup completion times

VTLs are typically much faster than tape and so from a pure backup window standpoint, a customer will typically see a substantial reduction in backup window when implementing a VTL. (SEPATON’s S2100-ES2 provides 300 MB/sec single stream performance!)

He then goes on to say that the real problem is creating offsite tapes and he talks about how it is a 3 step process. This is true, but the reality is that the same three step process is used for physical tape as well. Most customers maintain a copy of the data onsite in the physical tape library and copy at the DR site. Thus they need two copies even with physical tape, and the creation process is a three-step process for tape just like disk. ([during backup] backup server –> tape library, [during tape copy/vaulting] tape library –> backup server, backup server –> tape library

The question that he misses is “Would you rather use a tape-to-tape copy mechanism to create offsite tapes or a disk-to-tape copy process?” The answer is obvious. You will always want a disk-to-tape copy because of the improved performance and reliability.

VTLs extend time to DR

I am not sure what the point is here. Time to DR is the time it takes to complete the backup, copy the data to tape and get the data offsite. As mentioned above, VTL will dramatically outperform tape in the process of both backing up and copying data thus improving DR time.

He also talks about VTL replication. This is an entirely separate discussion and users need to review this in the context to their offsite requirements. One side note, is that SEPATON’s Site² technology integrates with the backup application so data is replicated in a manner that is dictated by the backup application’s policy engine.

Most VTL restores are not from disk

This is just not true. If you ask most users, they will tell you that something like 80% of their restore requests is for data that is less than 2 weeks old. The goal would then be to implement a VTL that allows for 2 weeks retention. If you do this then more than ¾ of your restores will come from disk. 90%+ of our customers are retaining at least two weeks of data on the VTL. Perhaps he is assuming that the VTL is tiny that you only retain a few days on disk? If so, then most restores would be from tape, but that is a very inaccurate assumption and does not match reality.

VTLs do not reduce tape media expenditures

The impact of VTL on tape media expenditures depends on how customers use tape. As mentioned previously, the vast majority of customers use VTL to replace tape assets that were kept onsite prior to VTL. In this scenario, they will recognize immediate savings because they can replace all of those onsite tapes with a VTL and will only need tape for offsite purposes.

VTLs offer inefficient deduplication

This is a ridiculous gross generalization. His lack of knowledge is further reflected in the following statements:

To compensate for their lack of value, VTLs have attempted to add data deduplication as a product feature
Again, he is showing his bias here and clearly he thinks that there is no value in VTL. Industry statistics and real world customer experiences indicates otherwise.

The challenge is, because most of these systems have deduplication as a post-process
The question of post-process vs inline is an entirely separate discussion than VTL or no VTL. There are VTL vendors who offer inline deduplication and those who offer post-process. Suggesting that somehow VTL requires post-process is just idiotic.

In short, this article reflects an extremely biased via of the world. The author seems pre-disposed to hate VTL and is trying to do everything he can to discredit the technology. The article shows a clear lack of understanding of VTL and makes me wonder about the original author’s real agenda. I think that the most realistic assumption is that he is being unduly influenced by a vendor (because he is being paid or is otherwise involved) and thus providing a biased opinion.

The Fallacy of Faster Tape

Tape is not dead!

Jay Livens — Thu, 21 Aug 2008 17:51:56 +0000

I am amazed when I hear some vendors aggressively promote that tape is dead. It seems that hyping the demise of tape is in vogue these days and the reality is quite different. Even so, there is no stopping them from sharing their message with anyone who will listen. If you ask large enterprises, many of them are looking at alternatives to tape, but telling them that tape is completely dead and that they should rip out all tape hardware is ludicrous. Ironically, this is the approach of some deduplication vendors. Jon Toigo states this succinctly in his blog.

The problem with tape is that it has become the whipping boy in many IT shops.
Courtesy: Drunken Data

The simple reality is that tape has been an important component of data protection for years and is likely to maintain a role far into the future. The reader should remember that in today’s highly regulated environments, companies often face strict requirements about data retention. For example, medical institutions can face some of the most stringent requirements:

HIPAA’s Privacy Rule, in effect since 2003 or 2004 depending on the size of the organization, requires confidentiality of patient records on paper and sets retention periods for some kinds of medical information, regardless of media. These retention requirements can stretch from birth to 21 years of age for pediatric records, or beyond the lifetime of the patient for other medical records.
Courtesy: Directory M

With this in mind, let’s look at the evolution of tape:

First there was tape…
Prior to the advent of low cost disk drives, tape was the primary method of data protection and backup applications were architected to support tape. This was the driver of the media management abilities that are so prevalent in today’s backup applications.

Backup was relatively simple in these days. There were limited regulations and companies had few cost effective options outside of tape.

Then there was low-cost disk…
As low-cost disk emerged, most backup ISV’s recognized the benefits of the new medium and added a disk backup option. It is important to note that most disk backup implementations were developed to augment, not replace tape and were not nearly as mature and reliable as their tape-based counterpart. The immaturity of disk support was not deemed to be a problem since customers were only expected to retain data for short periods of time on disk.

VTL’s emerged around the same time. They were created to leverage the robust support of tape in the backup application while providing the performance and reliability of disk. The technology also allowed customers to retain data on disk more effectively since it leveraged tape media management. In practice customers typically retained more data on VTL than the disk-as-disk option described above.

The inclusion of disk into the backup process added complexity since the administrator had to decide how much disk to purchase, how much data to retain on disk and how to configure the disk subsystem for optimal performance. In all cases, disk was used for retaining data for a relatively short period of time (few days to a few weeks) and tape was still used for long-term retention.

Then there was deduplication…
Deduplication changes the economics of disk-based data protection. It allows for a dramatic reduction in $/GB although and as posted before, it often leads to a reduction in restore performance. It also allows for more cost effective replication since you can often use the technology to power the replication of deduped data. (e.g. sending only net new data over the wire.)

The explosion of deduplication solutions has lead some deduplication vendors to say that tape is no longer necessary and that all data should be backed up to disk directly and then replicated to a remote site for DR purposes. Is this really a valid approach for all companies? In short, the answer is no.

Small businesses traditionally have much less investment in physical tape and their environments are relatively simple. Thus, they are often looking to reduce the dependence on tape since it adds management complexity. The tapeless message resonates most with these users.

The situation is completely different with enterprises. These companies have strict business and data protection requirements. For example, as quoted above, hospitals are required to retain some data over a patient’s life time. Enterprises are typically already using disk and have substantial investments in tape. If you ask them whether they are willing to instantly rip out tape and replace it with disk, the answer will uniformly be no. Their conservative nature, strict business requirements and massive investment in tape means that they are not looking to radically change their processes. They may have a long-term goal of going tapeless, but their implementation process will be gradual and some applications may never move away tape.

In summary, saying “Tape is dead” is a generalization that only applies to a small subset of the market. In the enterprise, where SEPATON participates, companies are looking at VTL deduplication solutions as a way to enhance data protection, improve their current tape operation and provide a future path to reduce tape usage. (On a side note, it is ironic that the guy posting this works for a company whose name is “No Tapes” spelled backwards!) In general, I believe that tape is similar to mainframe computers. People have been forecasting the death of mainframes for the last 17 years (See this article from IBM), but the business keeps going and so it is with tape. And so if you run into a vendor telling you that tape is dead and I suggest that you answer, “Yup, just like mainframes are dead!”