Figure 8 - uploaded by William Sanders
Content may be subject to copyright.
Source publication
In this paper we present a framework for ana-lyzing the fault tolerance of deduplicated storage systems. We discuss methods for building models of deduplicated storage systems by analyzing empirical data on a file category basis. We provide an algorithm for generating component-based models from this information and a specification of the storage s...
Contexts in source publication
Context 1
... shown in Figure 8 we have a set of n model decompositions that form intervals defined by the times d 0 , d 1 , . . . , d n−1 during the period [t, t + l]. ...
Similar publications
Cloud Computing aims to provide reliable services within data centres that contain servers, storage and networks. The services are delivered to the users transparently without their need to know the details of the underlying software and hardware. One of the challenge of cloud computing is to ensure that the applications run without a hiatus in the...
The binary states, i.e., success or failed state assumptions used in conventional reliability are inappropriate for reliability analysis of complex industrial systems due to lack of sufficient probabilistic information. For large complex systems, the uncertainty of each individual parameter enhances the uncertainty of the system reliability. In thi...
Citations
... In order to store more and more data, currently distributed storage systems are usually used for storage [3,4]. With the increasing size of distributed storage systems, in order to save costs, storage nodes are cheaper and less reliable devices [5][6][7], which leads to node failures more and more frequently. ...
Disk reliability is a serious problem in the big data foundation environment. Although the reliability of disk drives has greatly improved over the past few years, they are still the most vulnerable core components in the server. If they fail, the result can be catastrophic: it can take some days to recover data, sometimes data lost forever. These are unacceptable for some important data. XOR parity is a typical method to generate reliability syndrome, thus improving the reliability of the data. In practice, we find that the data is still likely to be lost. In most storage systems reliability improvements are achieved through the allocation of additional disks in Redundant Arrays of Independent Disks (RAID), which will increase the hardware costs, thus it will be very difficult for cost-constrained environments. Therefore, how to improve the data integrity without raising the hardware cost has aroused much interest of big data researchers. This challenge is when creating non-traditional RAID geometries, care must be taken to respect data dependence relationships to ensure that the new RAID strategy improves reliability, which is a NP-hard problem. In this paper, we present an approach for characterizing these challenges using high-dimension variants of the n-queens problem that enables performable solutions via the SAT solver MiniSAT, and use the greedy algorithm to analyze the queen’s attack domain, as a basis for reliability syndrome generation. A large number of experiments show that the approach proposed in this paper is feasible in software-defined data centers and the performance of the algorithm can meet the current requirements of the big data environment.
... For example, some studies (e.g., [3], [6], [20]) add redundancy via replication or erasure coding to post-deduplication data for fault tolerance. Other studies (e.g., [15], [30], [31]) propose quantitative methods to evaluate deduplication storage reliability. However, there remain two key open reliability issues, which are further complicated by the data sharing nature of deduplication. ...
... Li et al. [15] propose combinatorial analysis to evaluate the probability of data loss of deduplication systems. Rozier et al. [30], [31] propose automata-based frameworks to quantitatively evaluate the reliability of deduplication systems under disk failures and sector errors. Our work complements the above studies by: (i) adopting more robust reliability metrics, (ii) focusing on primary storage workloads, and (iii) comparing the impact of loss variations and repair strategies on storage system reliability with and without deduplication. ...
... When creating non-traditional RAID geometries, care must be taken to respect data dependence relationships [24] to ensure that the new RAID strategy improves reliability. We consider two types of data dependence relationships; one resulting from pre-existing RAID groups, and the other from data deduplication [22]. ...
As data centers attempt to cope with the exponential growth of data, new techniques for intelligent, software-defined data centers (SDDC) are being developed to confront the scale and pace of changing resources and requirements. For cost-constrained environments, like those increasingly present in scientific research labs, SDDCs also present the possibility to provide better reliability and performability with no additional hardware through the use of dynamic syndrome allocation. To do so the middleware layers of SDDCs must be able to calculate and account for complex dependence relationships to determine an optimal data layout. This challenge is exacerbated by the growth of constraints on the dependence problem when available resources are both large (due to a higher number of syndromes that can be stored) and small (due to the lack of available space for syndrome allocation). We present a quantitative method for characterizing these challenges using an analysis of attack domains for high-dimension variants of the n-queens problem that enables performable solutions via the SMT solver Z3. We demonstrate correctness of our technique, and provide experimental evidence of its efficacy; our implementation is publicly available.
Data deduplication is a widely used technique to remove duplicate data to reduce the storage overhead. However, deduplication typically cannot eliminate the redundancy among nonidentical but similar data chunks. To reduce the storage overhead further, delta compression is often applied to compress the post-deduplication data. While the two techniques are effective in saving storage space, they introduce complex references among data chunks, which inevitably undermines the system reliability and introduces fragmentation that may degrade the restore performance. In this paper, we observe that the delta compressed chunks (DCCs) are much smaller than regular chunks (non-DCCs). Also, most fragmentation caused by the base chunk of DCCs remain fragmented in consecutive backups. Based on these observations, we introduce a framework called , which combines replication and erasure coding and uses History-aware Delta Selection to ensure high reliability and restore performance. Specifically, uses a delta-utilization-aware filter and a cooperative cache scheme (CCS) to maintain cache locality and avoid unnecessary container reads, respectively. Moreover, the system selectively performs delta compression by historical information to avoid cyclic fragmentation in consecutive backups. Experimental results based on four real-world datasets demonstrate that significantly improves the restore performance by 58.3%–76.7% with a low storage overhead.
The reliability issue in deduplication-based storage systems has not received adequate attention. This paper proposes a Per-File Parity (short for PFP) scheme to improve the reliability of deduplication-based storage systems. PFP computes the XOR parity within parity groups of data chunks of each file after the chunking process but before the data chunks are deduplicated. Therefore, PFP can provide parity redundancy protection for all files by intra-file recovery and a higher-level protection for data chunks with high reference counts by inter-file recovery. Our reliability analysis and extensive data-driven, failure-injection based experiments conducted on a prototype implementation of PFP show that PFP significantly outperforms the existing redundancy solutions, DTR and RCR, in system reliability, tolerating multiple data chunk failures and guaranteeing file availability upon multiple data chunk failures. Moreover, a performance evaluation shows that PFP only incurs an average of 5.7% performance degradation to the deduplication-based storage system.
Deduplication has been widely used to improve storage efficiency in modern primary and secondary storage systems, yet how deduplication fundamentally affects storage system reliability remains debatable. This paper aims to analyze and compare storage system reliability with and without deduplication in primary workloads using public file system snapshots from two research groups. We first study the redundancy characteristics of the file system snapshots. We then propose a trace-driven, deduplication-aware simulation framework to analyze data loss in both chunk and file levels due to sector errors and whole-disk failures. Compared to without deduplication, our analysis shows that deduplication consistently reduces the damage of sector errors due to intra-file redundancy elimination, but potentially increases the damages of whole-disk failures if the highly referenced chunks are not carefully placed on disk. To improve reliability, we examine a deliberate copy technique that stores and repairs first the most referenced chunks in a small dedicated physical area (e.g., 1% of the physical capacity), and demonstrate its effectiveness through our simulation framework. IEEE
With the explosive growth in data volume, the I/O bottleneck has become an increasingly daunting challenge for big data analytics in the Cloud. Recent studies have shown that moderate to high data redundancy clearly exists in primary storage systems in the Cloud. Our experimental studies reveal that data redundancy exhibits a much higher level of intensity on the I/O path than that on disks due to relatively high temporal access locality associated with small I/O requests to redundant data. Moreover, directly applying data deduplication to primary storage systems in the Cloud will likely cause space contention in memory and data fragmentation on disks. Based on these observations, we propose a performance-oriented I/O deduplication, called POD, rather than a capacity-oriented I/O deduplication, exemplified by iDedup, to improve the I/O performance of primary storage systems in the Cloud without sacrificing capacity savings of the latter. POD takes a two-pronged approach to improving the performance of primary storage systems and minimizing performance overhead of deduplication, namely, a request-based selective deduplication technique, called Select-Dedupe, to alleviate the data fragmentation and an adaptive memory management scheme, called iCache, to ease the memory contention between the bursty read traffic and the bursty write traffic. We have implemented a prototype of POD as a module in the Linux operating system. The experiments conducted on our lightweight prototype implementation of POD show that POD significantly outperforms iDedup in the I/O performance measure by up to 87.9 percent with an average of 58.8 percent. Moreover, our evaluation results also show that POD achieves comparable or better capacity savings than iDedup.
As both the amount of data to be stored and the rate of data production grows, data center designers and operators face the challenge of planning and managing systems whose characteristics, workloads, and performance, availability, and reliability goals change rapidly. As we move towards software-defined data centers (SDDCs) the ability to reconfigure and adapt our solutions is increasing, but to take full advantage of that increase we must design smarter, more intelligent systems that are aware of how they are being used and able to deliver accurate predictions of their characteristics, workloads, and goals. In this paper we propose a novel algorithm for use in an intelligent, user-aware SDDC which performs run-time analysis of user storage system activity in a manner that has a minimal impact on performance and provides accurate estimations of future user activity.
Our algorithm can produce both generalized models, and specific models, depending on the parameters used. Our algorithms are efficient, and have low overhead, making them ideal to use to add intelligence to SDDCs and build intelligent storage systems. We use our algorithm to analyze actual data from two real systems, monitoring user activity two and three times a day for each system respectively, over a period of roughly two years, for almost 500 distinct users.
