Detecting subtle facial expressions using the Euler motion magnification method

Master's Thesis of Faculty of Technology and Engineering

Computer Department

Abstract:

In this thesis, a new method is presented to detect the subtle emotional states of the face. Euler's video zoom method has been used to reveal subtle facial movements. Euler's video zoom method enhances the subtle movements of signals (color change or transition motion) by temporal and spatial processing. The input of this method is videos of people's faces and the result is a sequence of enlarged images. Then we have examined the type of emotional state of the sequence of enlarged images using special faces. In this research, the experiment was conducted on 164 videos of 16 people in the database of subtle involuntary emotions. At first, we have examined the detection of changes in the face when subtle emotional states occur. The change in the face is detected on average by 65.19% when the negative state occurs, 68.29% when the positive state occurs, and 69.75% when the surprised state occurs. We have also evaluated the recognition and separation of two positive and negative states in the sequence of images. As the results show, the overall detection accuracy between negative and positive states using Euler video magnification is 66.66%. While without using this magnification method, the overall detection is reduced to 50.00%. Therefore, the use of Euler video magnification in retrieving imperceptible movements of the face when an emotion occurs increases the performance of the subtle emotional states detection method based on special faces as well as change detection.

Key words: detection of subtle emotional states of the face, facial change detection, Euler video magnification method, special faces method

Chapter 1

Introduction

1 Preface

Change detection is one of the new researches in the field of image processing and machine vision and has been growing in recent years. Change detection is a time segmentation process that is performed on images and videos taken at different times. Image change detection techniques play a fundamental role in video surveillance, medical imaging, underwater imaging, and traffic monitoring for automatic detection of moving objects and targets. One of the important applications in this field is the detection of changes in the face when emotions appear, which plays an important role in human communication. Therefore, recognition of facial emotional expressions (FER[1]) has not only been studied by many psychological researchers, but has also been proposed in the field of computer science. Computer scientists have made many efforts to identify and recognize real human facial expressions.

Facial expressions play an essential role in verbal and non-verbal communication and conveying opinions between people. Facial expressions express the actions and reactions that humans show towards their surroundings. Recognition of facial expressions can be useful in many research and application fields, including lip reading, video conferencing, robotics, psychological studies, etc.

In the field of robotics, there are humanoid robots that interact with the user and adapt to their mood. Also, in the field of psychology, tests have been designed in the field of therapy, by showing pictures to the patient, the reactions on his face are checked and the mental state of the person is analyzed. The discussion of simulating the emotional states of the face is also important in visual and cartoon animations. These cases and various other applications have led to many researches in this field. 1-2 Statement of the problem of detecting changes in the image In this section, the problem of detecting changes is presented in this way. The ultimate goal of detecting changes is to identify the set of pixels connected to each other in one frame that are different from the next frame. The set of pixels contains the change mask. The purpose of this problem is to identify this mask and segment the background.

This mask will be a combination of several basic factors, including the movement of the object towards the viewer, the change in the shape of objects and their appearance, and the disappearance of objects. Also, objects can change with different light intensity or color. The main challenge is that the change mask should not contain unknown and incorrect change forms caused by camera movement, light changes and weather conditions..

In order to detect the change, the M images taken by the camera every second are displayed as each pixel has coordinates and color. The values ??of k and l depend on the image type. For example, it is used for grayscale images and for RGB images. It is also considered for aerial images or satellite surveillance, or for medical or biological information. The mathematical technique to detect the change of each pixel for the input image is as follows"Chhabra et al., 2009"

If there is a significant change in pixel x of I(x). A=

(1-1) (formulas are available in the main file)

The facial emotional state detection system usually consists of a sequential structure of processing blocks" Khatri et al., 2014". The main blocks of this structure include pre-processing [2], feature extraction [3], classification [4] and post-processing [5]. In the pre-processing stage, actions such as noise reduction, normalization with respect to brightness fluctuations or normalization of dimensions and accuracy of images are performed. Feature extraction is a very important step in face recognition. Feature extraction methods can be divided into two categories"Tian et al., 2005" feature-based methods[6], appearance-based methods[7]. In feature-based methods, the geometric features of the face such as the location and shape of the facial components (eyes, nose, mouth and eyebrows) are important. Face components or face features are extracted in the form of a feature vector that represents the geometry of the face. Experimental results have shown that emotional features are not always reliably recognized, the reasons for which can be attributed to image quality, light, etc. pointed out In the appearance-based methods, the skin texture is paid attention to and features are extracted with the help of wrinkles and furrows that appear in different parts of the face during emotional states. The appearance-based method is performed in the entire face space or in a specific area of ??it using optical flow [8] to calculate the movement of facial muscles "Tian et al., 2002". This method is suitable for low quality images. In some articles presented in this field, a combination of appearance-based methods and feature-based methods (hybrid methods) have been used for feature extraction. The features extracted in this step are used as input to a classifier system. The output of the classifiers is generally one of the 6 main states defined by Ekman, "Ekman, Friesen, 1987", i.e. happiness, sadness, surprise, anger, fear and disgust Figure 1-2. They have defined the movement unit set (AU[9]) by dividing each movement into a series of elementary movements. In this method, the movements of the facial parts are tracked. Then, their changes are categorized in the facial expression coding system (FACS[10]) and emotions are interpreted. The purpose of the post-processing stage is to improve the recognition accuracy.

4 Objectives of this model and instructions

Most studies of facial expressions have been conducted on databases where all people pretended to express emotions. A number of studies have been conducted to aid in the detection of very subtle emotions in computer vision (Pfister et al., 2011). For example, the subtle movements that occur on people's faces when they feel emotions are not visible to the eye. Therefore, the purpose of this research is to use Euler's video zooming method to detect partial human emotional states in order to show the subtle movements of the face when emotions appear in a person. Then we determine the type of emotional state of the resulting data using the method of special faces [11] and check the effectiveness of Euler's video zooming method. It is clear that fine-grained emotion datasets [12] are needed to achieve more accurate and practical results. Therefore, the data collected by Li et al., 2013 have been used.

(Images are available in the main file)

The structure of the thesis is organized as follows. Also, the types of databases will be introduced and explained in this context.

In chapter 4, the method of special faces is analyzed.

In chapter 5, Euler's video zooming method is described.