Literature Survey on Super Resolution and its Challenges Murali Krishna AtmakuriAsst
Literature Survey on Super Resolution and its Challenges
Murali Krishna AtmakuriAsst. Prof,Dept. of ECE
RVR&JCCE,[email protected]
Anil Kumar KattaAsst. Prof,Dept. of ECE
RVR&JCCE,[email protected] Prasad
professor&Head, Dept. of ECE
GEC,[email protected]
Abstract- In the recent trends, image processing field became an interesting area for researchers due to vast advancements over the few decades. Now-a-days one of the most recent and important trend of image processing is Super Resolution. Super resolution utilizes the features of reconstruction; reconstruction is a kind of producing high spatial image from one or more low resolution images. Super resolution combines the non-redundant information of low-resolution images to develop high-resolution images. This article envisages the recent advances in super-resolution techniques and provides its advantages and disadvantages. This article also explains the challenges of super resolution and the scope further research studies.
Index Terms- Image Resolution, Super Resolution, Interpolation, Wavelet Transform, Learning, Reconstruction.
INTRODUCTION
Vision is one of the most important of five senses, so images not only play the important role but also used to make decisions based on human perception. In order to obtain better human perception, it is important to use high resolution images 1-6. High resolution images are used in various kinds of applications, some of the examples are as follows: Military and Civilian 7-10. The recent trends in image and video sensing have been intensified by the expectations of the user on visual quality of captured-data 11, 12. High quality visual captured-data can be obtained with the help of high resolution cameras. The limitation of using high resolution cameras are: Expensive, Need high power, Need high memory size and limited band width even though the high resolution cameras used, sometimes it is not possible to obtain high resolution image 13. To circumvent aforementioned limitations super resolution found to be an effective solution. In variety of digital imaging applications, high resolution images are used. Image resolution describes the details of image and higher in image resolution gives more details about the image 14-18. Super resolution became an interesting area for researcher since more the resolution gives more data about image 19-22. Resolution is generally determined by pixel density 23.
Figure 1 presents the generic image acquisition process, where diverse factors affect the image quality like: Over the air (OTA), Charge-couple device (CCD), Pre-processors and Environment. Optical Blur is a non-symmetric design of the lens and an aperture before or behind the optic center of the lens lead to
image distortions. Motions blur results when the image being recorded changes during the recording of a single frame, either due to rapid movement or long exposure. Noise in an image is an undesirable by-product of image capture that adds spurious and extraneous information.
One way of producing a high resolution (HR) image, is by installing a high resolution sensor. But it is not very feasible to do so. It results in increase of a cost as well as increase in power consumption. A simple example of this is satellite imaging system or a medical imaging system, where it is infeasible to use a high resolution sensor. So, to come over these, post processing is required to develop a better resolved image that holds more information. One of the promising approaches for this is signal processing techniques to obtain HR image from multiple low resolution (LR) images. Now-a-days such approach is more active in research area, and is called super resolution or Resolution Enhancement 21.
IMAGE sUPER RESOLUTION
Super resolution can be acquired either by processing multiple low resolved images as input and generating a high detail containing a single super resolved image as output or enhancing the details in a single low resolved image and generating a high resolved image for analysis. In SR from multiple LR images, it is a construction of HR image from several LR images, thereby increasing the high frequency components. The basic idea behind this is to combine non-repetitive information contained by multiple LR images.
Fig. 1. Generic image acquisition system
510540059055004067175266700Restoration for noise and blur removal
00Restoration for noise and blur removal
34575756000750781050266700Registration or Motion estimation
00Registration or Motion estimation
2381250266700Interpolation onto a high resolution grid
00Interpolation onto a high resolution grid
17716506191250666754286250066675638175006667584772500Input: LR IMAGES Output: SR IMAGES
PAGE * MERGEFORMAT 2
Fig. 2. Basic super resolution reconstruction stages
The main benefit of SR approach is that, a HR image can be obtained even with the existing LR imaging with lower cost and less power consumption.
Usually, a super resolution method consists of the following basic processing steps: (1) Registration, (2) Interpolation and (3) De-blurring or noise removal.
Image registration is the process of overlapping more than one images of the same scene which has been taken from different angles by the sensors. In registration two or more images are align geometrically to obtain the information through image fusion or change detection.
Interpolation is a process of estimating the intermediate pixels between the pixel values. When any image is converted from LR to HR, intermediate gaps are introduced and these values have to be estimated and filled with interpolation process.
As the process of interpolation introduces some artifacts the resultant image will be blurred or noisy. Through different filters and techniques noise will be removed and finally a super resolved image is generated.
Approaches to Super resolution
Super-resolution techniques can be classified as (1) Frequency domain approach and (2) Spatial domain approach.
Frequency domain approach
Transform the LR image into Discrete Fourier Transform (DFT) domain and combined them according to the relationship between the aliased DFT coefficients of the observed LR images and that of the unknown high-resolution image. The combined data are then transformed back to the spatial domain where the new image could have a higher resolution than that of the input images 3. The principles of frequency domain approach are as follows: i) what is the shift property of the Fourier transform, ii) The aliasing association between the continuous Fourier transform (CFT) of an original HR image and the discrete Fourier transform (DFT) of observed LR images, iii) the supposition that an original HR image is band limited.
Frequency domain approach has some benefits like it is spontaneous way to enhance the details by extrapolating the high frequency information presented in LR images. As well it has lower computational complexity. But the disadvantage is that is incapable of handling the real-world applications.
Spatial Domain Approach
The frequency domain approach has certain drawbacks like it limits the inter-frame motion to be translational. As well it is very difficult in frequency domain to use the prior knowledge. As the main problem is ill-posed image in SR, prior knowledge is required to overcome this. The main benefit of spatial domain is the support for unbind motion between frames and prior knowledge availability for solving the problems. Some of the methods are interpolation, iterative back projection and projection onto convex.
3.2.1. Interpolation
Interpolation is the process of transferring image from one resolution to another without losing image quality. In Image processing field, image interpolation is very important function for doing zooming, enhancement of image, resizing and many more. Most common interpolation techniques are nearest neighbor, bilinear and cubic convolution. Digital image is a signal, spatially varying in two dimensions. This signal is sampled and quantized to get values. All these values called pixels of image. When we increase the resolution of image from low to high, it is called up-sampling or up-scaling while reverse is called down sampling or down scaling.
Interpolation is of three types: (i) Bi-linear Interpolation: In Bi-linear interpolated point is filled with four closest pixel’s weighted average. Bi-linear interpolation is recommended for continuous data like elevation and raw slope values. (ii) Bi-cubic Interpolation: Bi-cubic interpolation is recommended for smoothing continuous data, but this incurs a processing performance overhead and (iii) Nearest Neighbor Interpolation: In this method, nearest value is copied for interpolation and this technique has less computational complexity. Nearest neighbor interpolation is recommended for categorical data such as land use classification.
3.2.2. Iterative Back Projection (IBP)
In IBP approach HR image is estimated by back projecting the difference between the simulated LR image and captured LR on interpolated image. This iterative process of SR does iterations until the minimization of the cost function is achieved.
3.2.3. Classical Multi-Image Super Resolution
In the classical multi-image SR, a set of LR images of the same scene is taken. If enough LR images became available then the equation is determined and a SR image is reconstructed. The assumption here is that the two or more LR images should contain distinguishable features. Because of these, practically it helps very less in improvement of image resolution, if distinguishable features in LR images are less.
3.2.4. Example Based Super Resolution
In Example- Based approach, the same rule is applied. This approach is useful when only single LR image is available. In this approach, the image has small patches that redundantly reappear, both within the scale as well as across the scale. Each LR patch in an image is replaced by its corresponding HR patch to generate the SR image. Here assumption is that, the image should have enough HR patches for the correspondence LR patches.
3.2.5. Learning Based Super Resolution
It is a concept of machine learning, where the machine is trained to classify LR and its corresponding HR patches. In this approach, both LR and HR patches are divided into different classes. By doing so, the number of comparison reduces, as it has to compare LR with only HR patches. For an image if it is an edge-area of the LR, the routine example-based image SR algorithm can be used to implement the local and fine SR. For the flat regions of the low-resolution, only interpolation algorithm is used for super-resolution. The performance of learning based super-resolution depends on HR patches retrieved from the training data for an input LR patch.
Table 1. Comparison among various super resolution approaches 21.
Categorization Description Disadvantages
Interpolation Based Different interpolation techniques used Over-smooth jagged artifacts
Reconstruction Based Reconstruction constraint and image prior Ringing artifacts, imposing additional prior
Learning Based Learning high frequency details from training set High frequency artifacts, relying train set
Challenging issues of Super Resolution
In practice developing super resolution image, there are several challenging issues. Some of the challenging issues are mentioned below:
Image Registration
In an image, image registration is a well-known problem known by the name of ill-posed image. Image registration becomes more and more difficult when observed LR image is having very high aliasing effects. The registration error increases with decrease in the resolution of observed image. The degradation caused by these registration errors affects the quality of an image resolution more than that of interpolation 21. List may be presented with each item marked by bullets and numbers.
Computational Efficiency
Real time application is always requires good efficiency. As there are large numbers of unknowns in reconstructing super resolution images, matrix manipulation increases.
Robustness
Super resolution techniques are defense-less to the presence of outliers due to motion errors, inaccurate blur models, noise, moving objects, motion blur etc. These effects are not easy to estimate which are not acceptable in many applications.
Conclusion
This article provides literature review about various techniques used to achieve super resolution image with the help of single image or multiple low resolution images. In this article, interpolation based, reconstruction based and learning based techniques for super resolution are studied. This article gives detailed comparison of several super resolution approaches.
The super resolution applications like medical sciences and satellite imaging has been presented. The future scope of the super resolution is developing new methods by integrating or extending the existing methods to address their challenges. Finally images with super resolution can provide more details as compared to low resolution images with low cost.
References
M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda. Infrared im-age registration and high-resolution reconstruction using multiple translationally shifted aliased video frames. IEEE Transactions on Instrumentation and Measurement, 49(5):915-923, 2000.
S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167-1183, 2002.
O. Bowen and C. S. Bouganis. Real-time image super resolution using an FPGA. In International Conference on Field Programmable Logic and Applications, pages 89-94, 2008.
A. Buades, B. Coll, and J. M. More. A non-local algorithm for image denoising. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 60-65, 2005.
J. Chung, E. Haber, and J. Nagy. Numerical methods for coupled super-resolution. Inverse Problems, 22(4):1261-1272, 2006.
D. Datsenko and M. Elad. Example-based single document image super-resolution: a global MAP approach with outlier rejection. Multidimen-sional System and Signal Processing, 18(2-3):103-121, 2007.
D. L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289-1306, 2006.
D. L. Donoho. For most large underdetermined systems of linear equations, the minimal `1-norm near-solution approximates the spars-est near-solution. Communications on Pure and Applied Mathematics, 59(7):907-934, 2006.
X. Jia F. Li and D. Fraser. Universal HMT based super resolution for remote sensing images. In IEEE International Conference on Image Processing, pages 333-336, 2008.
S. Farsiu, D. Robinson, M. Elad, and P. Milanfar. Fast and robust multi-frame super-resolution. IEEE Transaction on Image Processing, 13(10):1327-1344, 2004.
P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar. Simul-taneous super-resolution and feature extraction for recognition of low-resolution faces. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1-8, 2008.
K. Jia and S. Gong. Generalized face super-resolution. IEEE Transactions on Image Processing, 17(6):873-886, 2008.
A. Levin, Y. Weiss, F. Durand, and W. Freeman. Understanding and evaluating blind deconvolution algorithms. In Proceedings of IEEE Com-puter Society Conference on Computer Vision and Pattern Recognition, pages 1964-1971, 2009.
Frank Lin, Clinton B. Fookes, Vinod Chandran, and Sridha Sridharan. Investigation into optical low super-resolution for surveillance applica-tions. In The Austrilian Pattern Recognition Society Worshop on Digital Image Computing, 2005.
K. Malczewski and R. Stasinski. Toeplitz-based iterative image fusion scheme for MRI. In IEEE International Conference on Image Processing, pages 341-344, 2008.
B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao. A computa-tionally efficient super-resolution algorithm for video processing using partition filters. IEEE Transactions on Circuits and Systems for Video Technology, 17(5):621-634, 2007.
N. Nguyen and P. Milanfar. An efficient wavelet-based algorithm for image super-resolution. In Proceedings of International Conference on Image Processing, volume 2, pages 351-354, 2000.
N. Nguyen, P. Milanfar, and G. H. Golub. A computationally e±cient image superresolution algorithm. IEEE Transactions on Image Process-ing, 10(5):573-583, 2001.
M. Protter and M. Elad. Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 18(8):1899-1904, 2009.
P. Hardeep, B. Prashant, Swadas, M. Joshi, A Survey on Techniques and Challenges in Image Super Resolution Reconstruction. International Journal of Computer Science and Mobile Computing, 2013.
T. sai and Huang. A survey on super-resolution imaging. Springer, 2010.
R. SudheerBabu, K. E. S. Murthy. A Survey on the Methods of Super-Resolution Image Reconstruction. International Journal of Computer Applications. 15(2), 2011.
A. J. Shah, B. S. Gupta. Image Super Resolution – A Survey. IEEE, 2012.
