Enhancing the resolution of a color image from a string of low resolution images

Abstract

In recent years, there have been extensive developments in the field of image sensors and digital imaging systems, but still theoretical and practical limitations affect the resolution of images taken with these cameras. Super resolution techniques have been developed in recent years to overcome these limitations. These techniques create a higher resolution image by using one or more low resolution images. Recent works in the field of meta-resolvability (often focused on grayscale images) have been done in order to reduce computational complexity and increase resistance to modeling errors and noise. On the other hand, several de-mosaicing methods have been proposed in order to reduce color artifacts, which is a result of using single CCD cameras.

In this thesis, using statistical methods in signal processing, we propose a robust framework for combining low-resolution images to create a high-resolution image. In this method, by using the error-resistant criterion in the objective function and adapting the estimation process for each low-resolution image according to the accuracy of the model parameters and its noise level, we have created a robust reconstruction. Also, by generalizing this method in the field of color, and integrating the image resolution and demosonic process, we were able to simultaneously de-mosaicize the image in addition to increasing the resolution of color images. The tests performed also confirm the good performance of the proposed algorithm against noise and error. Keywords: meta-resolvability, image registration, M-estimation, regulator, image de-mosaicing, color filter. 1 Introduction The use of films and images with high resolution is required in most electronic applications. The desire to use high-resolution images stems from two main areas: improving visual information for human interpretation; and help to understand automatic devices. Image resolution describes the details in the image. The higher the resolution, the more detailed the image. The resolution of a digital image can be classified in many different areas: pixel resolution, spatial resolution, spectral resolution, temporal resolution, and radiometric resolution [1]. In this thesis, topics are discussed in the field of spatial resolution.

Spatial resolution: A digital image is made of small image elements called pixels. Spatial resolution refers to the density of pixels in an image, and the measure of that pixel is per unit area. Figure 1-1 shows the classic test to determine the spatial resolution of an imaging system. The spatial resolution of the image is first limited by the imaging sensors or the image acquisition device. In a digital camera, imaging is not done on film but by a sensitive sensor (charge coupled device (CCD) [1] or complementary metal oxide semiconductor (CMOS) [2]). These sensors are usually arranged in a two-dimensional array to capture a two-dimensional image signal. First of all, the size of the sensor or equivalently the number of sensor elements per unit area determines the spatial resolution of the image. Sensors with higher density enable greater spatial resolution for the imaging system. An imaging system with insufficient detectors produces low-resolution images with blocky effects due to low spatial sampling frequency. Many efforts have been made to increase the resolution of digital images, which can be divided into two general parts: software and hardware. Figure 1-1 USAF 1951 resolution pattern, a classic test, which is used to determine the resolution of imaging systems and sensors [3]. Increased image resolution. In addition, as digital camera sensor cells get smaller, the amount of effective light received by each cell decreases; Of course, it is possible to increase the amount of effective light received by each sensor cell by creating a network of convex lenses on the upper layer of the sensor cells. However, due to the presence of a large number of sensor cells, the shock noise caused by switching off and on within this cellular network still exists and becomes an effective factor in reducing the quality of the final image [2].

     While image spatial resolution is limited by image sensors, image detail (high frequency bands) is also limited due to lens blur (related to the sensor's point spread function), lens aberration effects, aperture refraction, and optical blur due to motion. Therefore, the hardware method to achieve images with higher quality and resolution is very expensive and practically impossible to some extent, and it is usually not possible to exceed a certain limit due to the technical limitations in the integrated circuit manufacturing technology. In addition to cost, the resolution of a surveillance camera is also limited by the speed of the camera and storage hardware. In some other cases, such as satellite images, it is difficult to use high-resolution sensors due to its physical limitations.

Using software method is proposed to accept image failures and use signal processing in the post-processing of captured photos, in order to interact between computing costs and hardware costs. Software methods are economically viable and provide the possibility of producing a higher resolution image by the same low resolution digital imaging cameras.

One of the techniques proposed in the software dimension, in order to increase the quality of the image, both in terms of the number of pixels and in terms of reducing the amount of noise, is the technique of super resolution (SR) [3]. In terms of naming, this technique is called meta-resolution because we will be able to go beyond the capabilities of the imaging system, and it is mainly divided into two groups of learning-based methods and multi-frame reconstruction-based methods [4]. In learning-based methods, only a low-resolution image (LR) [4] is used to create a high-resolution image (HR) [5]. This approach is a subgroup of machine learning methods. Some suggested methods in this field are given in [4-10]. The next group is the methods based on multi-frame reconstruction, which our focus in this thesis is on this group of techniques.

In multi-frame decomposition techniques, combining several images with lower resolution produces a final image with higher resolution. This process restores the high frequency components and removes the damage caused by low resolution camera imaging. The main idea in multi-frame meta-decomposition techniques is to combine non-redundant information in low-resolution frames to produce a high-resolution image [3]. A method closely related to SR is the image interpolation approach, which can be used to increase image size. But, since no additional information is generated, the quality of single-image interpolation is very limited due to the ill-posed nature of the problem, and it cannot recover the missing frequency components. But in the context of SR, numerous low-resolution observations are available for reconstruction. The non-redundant information in these low-resolution images is typically caused by fractional pixel displacements that occur between these images. These sub-pixel displacements may occur due to uncontrolled motions between the imaging system and the scene, for example, object motion; Or due to controlled movements, such as a satellite imaging system in Earth orbit that is moving at a predefined speed and path.

Each low-resolution frame is a distorted view of the real scene. Meta-resolvability is only possible if there is motion within a fraction of a pixel between these low-resolution frames. Figure 2-1 shows a simple diagram describing the basic idea of ??SR reconstruction. In the imaging process, the camera captures several LR frames from the HR scene. These LR images have fractional pixel shifts relative to each other and are also sampled at a low rate. The construction of multi-frame SR techniques is the reverse of this process; Alignment of LR observations in a fraction of a pixel accuracy, and combining them into an HR image network (interpolation), which overcomes the limitations of camera imaging.

Figure 1-2 The basic idea of ??reconstructing metaresolvability from low-resolution frames. The relative movement of low-resolution frames as a fraction of a pixel helps in reconstructing the super-high definition image [3]. According to Figure 3-1(a), the scene consists of four high-resolution pixels. By moving a fraction of the controlled pixel, the imaginary camera, consisting of only one pixel, creates a train of images from this scene.