Figure 3 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Source publication
The demand for communicating large amounts of data in real-time has raised new challenges with implementing high-speed communication paths for high definition video and sensory data. It requires the implementation of high speed data paths based on hardware. Implementation difficulties have to be addressed by applying new techniques based on data-or...
Contexts in source publication
Context 1
... us see the example of compression and decompression operations depicted in Figures 3 and 4. Assuming that the input data stream for the compressor is "ABABCDACABEFDCAB", the compressor first reads the first two symbols "AB" and tries to match them with the table (Figure 3a). ...
Context 2
... us see the example of compression and decompression operations depicted in Figures 3 and 4. Assuming that the input data stream for the compressor is "ABABCDACABEFDCAB", the compressor first reads the first two symbols "AB" and tries to match them with the table (Figure 3a). However, it fails to match them and registers "A" and "B" as the s0 and s1 in the table. ...
Context 3
... the compressor reads a pair of symbols (again "AB"), it finds a match in the table. Figure 3b translates "AB" to "0". Subsequently the equivalent operations are performed. ...
Context 4
... the equivalent operations are performed. If the table becomes full (Figure 3c), the compressor decrements count(s) of all entries until any counts become zero. Here, three entries are invalidated from the table in the figure. ...
Context 5
... compressor will register a new entry in place of the invalidated entry corresponding to the smallest index of the table. Figure 3d shows that the compressor added a new entry after the invalidation. Finally, the original input data is compressed to "AB0CDAC0EFDC0". ...
Context 6
... output is "AB". The decompressor reads the next symbol "0" (Figure 3b). It matches the table entry. ...
Similar publications
Coding algorithms are generally aimed at minimizing the
output code length encoding speed once the code has been
designed. Moreover, most codes use a binary alphabet. This paper
examines other issues related to coding, such as additional
constraints imposed on the channel. Code generation will be
considered where there is a limit on code words. Lim...
Citations
... First, we consider the techniques of dynamic invalidation on LUTs and lazy compression [11] that eliminate stalls during the LUT invalidations. ...
In this chapter, we introduce aspects of applying data-compression techniques. First, we study the background of recent communication data paths. The focus of this chapter is a fast lossless data-compression mechanism that handles data streams completely. A data stream comprises continuous data with no termination of the massive data generated by sources such as movies and sensors. In this chapter, we introduce LCA-SLT and LCA-DLT, which accept the data streams, as well as several implementations of these stream-based compression techniques. We also show optimization techniques for optimal implementation in hardware.
... The digram coding uses a look-up table that is used to compress an original data with associating to the index. The methods, called LCA-DLT [1] and ASE coding [2], employ a fixed number of entries in the table and recycle the entries during the compression/decompression. LCA-DLT organizes a compressor/decompressor module that converts a pair of original data to/from an index of the look-up table. The modules can be connected to compress/decompress multiple data pairs. ...
... During the registration operation for a new entry to the table, any entry might not be invalidated. In this case, we can use the lazy compression mechanism [1] to skip the symbol pair. As explained above, LCA-DLT compresses data stream without stalling by using a fixed number of entries. ...
It is getting popular to implement an environment where communications are performed remotely among IoT edge devices, such as sensory devices and the cloud servers due to applying, for example, artificial intelligence algorithms to the system. In such situations that handle big data, lossless data compression is one of the solutions to reduce the big data. In particular, the stream-based data compression technology is focused on such systems to compress infinitely continuous data stream with very small delay. However, during the continuous data compression process, it is not able to insert an exception code among the compressed data without any additional mechanisms, such as data framing and the packeting technique, as used in networking technologies. The exception code indicates configurations for the compressor/decompressor and/or its peripheral logics. Then, it is used in real time for the configuration of parameters against those components. To implement the exception code, data compression algorithm must include a mechanism to distinguish original data before compression and the exception code clearly. However, the conventional algorithms do not include such mechanism. This paper proposes novel methods to implement the exception code in data compression that uses look-up table, called the exception symbol. Additionally, we describe implementation details of the method by applying it to algorithms of stream-based data compression. Because some of the proposed mechanisms need to reserve entries in the table, we also discuss the effect against data compression performance according to experimental evaluations.
... On the other hand, hardware-based approaches, such as nx842 [16] and LCA-DLT [17], are proposed. The nx842 is used in POWER processors to compress data stream in the memory bus. ...
... Here let us describe four evaluations: the entropy coding ability, the performance effect of the near neighbor entry exchange, the optimization effects, and the hardware resource impacts. To compare the compression ratio, we used the entropy coding by the Shannon's information entropy based on eight bit and 16 bit symbol, LZ77 [18], the Deflate of 7-zip [19], and LCA-DLT [17] of connecting two compression modules for eight-bit symbol. The evaluation in this section uses the ASCII data sequences of [20] (Gene DNA sequences, Protein sequences, English texts, MIDI pitch values, XML, Linux source code). ...
Toward strong demand for very high-speed I/O for processors, physical performance growth of hardware I/O speed was drastically increased in this decade. However, the recent Big Data applications still demand the larger I/O bandwidth and the lower latency for the speed. Because the current I/O performance does not improve so drastically, it is the time to consider another way to increase it. To overcome this challenge, we focus on lossless data compression technology to decrease the amount of data itself in the data communication path. The recent Big Data applications treat data stream that flows continuously and never allow stalling processing due to the high speed. Therefore, an elegant hardware-based data compression technology is demanded. This paper proposes a novel lossless data compression, called ASE coding. It encodes streaming data by applying the entropy coding approach. ASE coding instantly assigns the fewest bits to the corresponding compressed data according to the number of occupied entries in a look-up table. This paper describes the detailed mechanism of ASE coding. Furthermore, the paper demonstrates performance evaluations to promise that ASE coding adaptively shrinks streaming data and also works on a small amount of hardware resources without stalling or buffering any part of data stream.
... It truly compresses data stream by calculating instantaneous entropy from the number of occupied entries in the look-up table at the timing when a symbol is received by the compressor. It works faster and provides 10 times smaller implementation on FPGA than the previous version of our stream-based data compression method [12]. However, those stream-based method needs to adjust parameters to achieve the best compression ratio depending on the data types. ...
... This allows to dynamically replace table entries depending on data entropy of data stream. We have proposed an effective hardware implementation of this mechanism [12], [24]. However, we still had a problem that the symbol is not compressed to smaller than the index width of the table. ...
As speed of communication data path is drastically improved in this decade due to the high data rate, evolutional technology is demanded to address the fast communication implementation. In this paper, we focus on data compression technology to speed up the communication data path. We have proposed a stream-based data compression called ASE coding. It compresses data stream based on the instantaneous data entropy without buffering and stalling for the compression processes. It is also suitable for hardware implementation. However, the stream-based data compression works heuristically with sensitive parameters that affect to the data compression ratio. If the parameters are statically configured, it does not follow the dynamic data entropy, and thus, the data compression performance becomes unstable. In this paper, we will disseminate the parameters, discuss the behaviors of those parameters and propose its autonomous adjustment methods. We will also propose adjustment algorithms for those parameters that follow the data entropy of the input data stream autonomously. Through experimental evaluations applying the algorithms, we will confirm the parameters are adjusted with depending on the data entropy in the data stream. And then, the compression ratio becomes stable as the compressor exploits the minimal entropy adaptively.
... Research on implementation of online grammar compression algorithms on small-sized hardware is ongoing. Yamagiwa et al. implemented FPGA-based lossless compression hardware [47,76]. They utilized a compact-memory grammar compression algorithm called LCA-DLT, which is another variation of LCA [48]. ...
A data structure is called succinct if its asymptotical space requirement matches the original data size. The development of succinct data structures is an important factor to deal with the explosively increasing big data. Moreover, wider variations of big data have been produced in various fields recently and there is a substantial need for the development of more application-specific succinct data structures. In this study, we review the recently proposed application-oriented succinct data structures motivated by big data applications in three different fields: privacy-preserving computation in cryptography, genome assembly in bioinformatics, and work space reduction for compressed communications.
... Universal coding requires a large amount of memory for hardware implementation. However, it is not easy to implement these algorithms in hardware [17]. The compute unified device architecture (CUDA) launched by NVIDIA can combine CPU with GPU to realize these algorithms [18]. ...
In motion estimation, the search regions of two adjacent current blocks have overlapping data. In view of this, the paper proposes a data reuse method for fast search motion estimation. The method reuses the overlapping data between the search areas of two adjacent blocks. The overlapping data are further divided into two parts, a definite data reuse area and a possible data reuse area. With this method, the memory access time of the algorithm is reduced, and the performance of the algorithm is further improved. And the proposed reuse method can effectively reduce the loading of redundant data. A typical fast search algorithm, diamond search, is used as a case study to verify the effectiveness of the method. The method is implemented on GPU platform. The experimental results show that the data reuse method can reduce the running time by 40%–60% compared with the algorithm of no use data reuse.
... Universal coding requires a large amount of memory for hardware implementation. However, it is not easy to implement these algorithms in hardware [17]. The compute unified device architecture (CUDA) launched by NVIDIA can combine CPU with GPU to realize these algorithms [18]. ...
Motion estimation (ME) is a time-consuming algorithm to find a matching block in the search area for video applications such as video compression. Motion estimation algorithm includes full search algorithm and fast search algorithm. If the width of the motion estimation algorithm search area is twice the size of the block, half of the search area for adjacent blocks overlaps. In view of this, the paper proposes a search area data reuse method for fast search motion estimation. With the proposed data reuse implementation, only the first block needs to read the data of the entire search area from the off-chip memory. The other blocks only need to read half the original search area. In this way, time of access to external memory is reduced, and running time of the algorithm is also decreased. Experimental results of diamond search (DS) show that the search area data reuse methods can reduce the running time by 40% to 60% compared with the algorithm of no use data reuse, and it can also reduce the power consumption by 62% to 73%. Compared with other methods in the literature, the proposed method also performs better on running time and power consumption.
This open access book gives an overview of cutting-edge work on a new paradigm called the “sublinear computation paradigm,” which was proposed in the large multiyear academic research project “Foundations of Innovative Algorithms for Big Data.” That project ran from October 2014 to March 2020, in Japan. To handle the unprecedented explosion of big data sets in research, industry, and other areas of society, there is an urgent need to develop novel methods and approaches for big data analysis. To meet this need, innovative changes in algorithm theory for big data are being pursued. For example, polynomial-time algorithms have thus far been regarded as “fast,” but if a quadratic-time algorithm is applied to a petabyte-scale or larger big data set, problems are encountered in terms of computational resources or running time. To deal with this critical computational and algorithmic bottleneck, linear, sublinear, and constant time algorithms are required.
The sublinear computation paradigm is proposed here in order to support innovation in the big data era. A foundation of innovative algorithms has been created by developing computational procedures, data structures, and modelling techniques for big data. The project is organized into three teams that focus on sublinear algorithms, sublinear data structures, and sublinear modelling. The work has provided high-level academic research results of strong computational and algorithmic interest, which are presented in this book.
The book consists of five parts: Part I, which consists of a single chapter on the concept of the sublinear computation paradigm; Parts II, III, and IV review results on sublinear algorithms, sublinear data structures, and sublinear modelling, respectively; Part V presents application results. The information presented here will inspire the researchers who work in the field of modern algorithms.
