Enhancing the resolution of a color image from a string of low resolution images

Senior Thesis for Master's Degree

Abstract

In recent years, there have been vast advances in the field of image sensors and digital imaging systems, but still theoretical and practical limitations affect the resolution of images taken with these cameras. Super resolution techniques have been developed in recent years to overcome these limitations. These techniques create a higher resolution image by using one or more low resolution images. Recent works in the field of meta-resolvability (often focused on grayscale images) have been done in order to reduce computational complexity and increase resistance to modeling errors and noise. On the other hand, several de-mosaicing methods have been proposed in order to reduce color artifacts, which is a result of using single CCD cameras.

In this thesis, using statistical methods in signal processing, we propose a robust framework for combining low-resolution images to create a high-resolution image. In this method, by using the error-resistant criterion in the objective function and adapting the estimation process for each low-resolution image according to the accuracy of the model parameters and its noise level, we have created a robust reconstruction. Also, by generalizing this method in the field of color, and integrating the image resolution and demosonic process, we were able to simultaneously de-mosaicize the image in addition to increasing the resolution of color images. The tests performed also confirm the good performance of the proposed algorithm against noise and error.

Keywords:

Super resolution, image registration, M-estimation, adjuster, image demosaicing, color filter.

1 Introduction

The use of videos and images with high resolution is required in most electronic applications. The desire to use high-resolution images stems from two main areas: improving visual information for human interpretation; and help to understand automatic devices. Image resolution describes the details in the image. The higher the resolution, the more detailed the image. The resolution of a digital image can be classified in many different areas: pixel resolution, spatial resolution, spectral resolution, temporal resolution, and radiometric resolution [1]. In this thesis, topics are raised in the field of spatial resolution.

     Spatial resolution: A digital image is made of small image elements called pixels. Spatial resolution refers to the density of pixels in an image, and the measure of that pixel per unit area.

Figure 1-1 shows the classic test to determine the spatial resolution of an imaging system. The spatial resolution of the image is first limited by the imaging sensors or the image acquisition device. In a digital camera, imaging is not done on film but by a sensitive sensor (charge coupled device (CCD) [1] or complementary metal oxide semiconductor (CMOS) [2]). These sensors are usually arranged in a two-dimensional array to capture a two-dimensional image signal. First of all, the size of the sensor or equivalently the number of sensor elements per unit area determines the spatial resolution of the image. Sensors with higher density enable greater spatial resolution for the imaging system. An imaging system with insufficient detectors produces low-resolution images with blocky effects due to low spatial sampling frequency. Many efforts have been made to increase the resolution of digital images, which can be divided into two general parts: software and hardware. Figure 1-1 USAF 1951 resolution pattern, a classic test, which is used to determine the resolution of imaging systems and sensors [3]. In the hardware part, by enriching the number of pixels on the On the sensors of digital cameras, it is possible to increase the resolution of the image per unit area.In addition, as digital camera sensor cells get smaller, the amount of effective light received by each cell decreases; Of course, it is possible to increase the amount of effective light received by each sensor cell by creating a network of convex lenses on the upper layer of the sensor cells. But due to the existence of a large number of sensor cells, the impact noise caused by the interruption and disconnection of the current within this cellular network still exists and becomes an effective factor in reducing the quality of the final image [2]. While the spatial resolution of the image is limited by the image sensors, the details of the image (high frequency bands) are also due to the blurring of the lens (related to the point spreader function of the sensor), the effects of lens deviation, aperture refraction and optical blur due to movement. are limited Therefore, the hardware method to achieve images with higher quality and resolution is very expensive and practically impossible to some extent, and it is usually not possible to exceed a certain limit due to the technical limitations in the integrated circuit manufacturing technology. In addition to cost, the resolution of a surveillance camera is also limited by the speed of the camera and storage hardware. In some other cases, such as satellite images, it is difficult to use high-resolution sensors due to its physical limitations.

The use of software methods is proposed to accept image failures and use signal processing in the post-processing of captured images in order to interact between computing costs and hardware costs. Software methods are economically viable and provide the possibility of producing a higher resolution image by the same low resolution digital imaging cameras.

One ??of the techniques proposed in the software aspect, in order to increase the quality of the image both in terms of the number of pixels and in terms of reducing the amount of noise, is the technique of meta resolution (SR)[3]. In terms of naming, this technique is called meta-resolution because we will be able to go beyond the capabilities of the imaging system, and it is mainly divided into two groups of learning-based methods and multi-frame reconstruction-based methods [4]. In learning-based methods, only a low-resolution image (LR) [4] is used to create a high-resolution image (HR) [5]. This approach is a subgroup of machine learning methods. Some suggested methods in this field are given in [4-10]. The next group is the methods based on multi-frame reconstruction, which our focus in this thesis is on this category of techniques.

In multi-frame decomposition techniques, combining several images with lower resolution produces a final image with higher resolution. This process restores the high frequency components and removes the damage caused by low resolution camera imaging. The main idea in multi-frame meta-decomposition techniques is to combine non-redundant information in low-resolution frames to produce a high-resolution image [3]. A method closely related to SR is the image interpolation approach, which can be used to increase image size. But, since no additional information is generated, the quality of single-image interpolation is very limited due to the ill-posed nature of the problem, and it cannot recover the missing frequency components. But in the context of SR, numerous low-resolution observations are available for reconstruction. The non-redundant information in these low-resolution images is typically caused by fractional pixel displacements that occur between these images. These sub-pixel displacements may occur due to uncontrolled motions between the imaging system and the scene, for example, object motion; Or due to controlled movements, such as a satellite imaging system in Earth orbit that is moving at a predefined speed and path. Meta-resolvability is only possible if there is movement within a fraction of a pixel unit between these low-resolution frames. Image super-resolution (SR) reconstruction is the process of generating an image at a higher spatial resolution by using one or more low-resolution (LR) inputs from a scene.